Category Archives: Infrastructure

All things to do with infrastructure such as server software setup and configuration, network topology etc

Hyper-V Manager – Connect to VM CredSSP error

[Window Title]
Remote Desktop Connection

[Content]
An authentication error has occurred.
The function requested is not supported

Remote computer:
This could be due to CredSSP encryption oracle remediation.
For more information, see https://go.microsoft.com/fwlink/?linkid=866660

The most annoying part about this was everything was fine

So I did the usual checks

1. Make sure the Windows Remote Management (WS-Management) Service is running

2. verify that CredSSP is enabled

3. Review the current settings

4. Make sure delegation is allowed from the host hyper-v server

Everything looked ok

Finally I stumbled on this MS article on credssp remediation error when RDP

Made the necessary registry key and value here
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\CredSSP\Parameters
“AllowEncryptionOracle”=dword:00000002

There is no reboot required, it worked immediately.
Yes you may now be vulnerable to the encryption oracle attack see Microsoft here

Windows HyperV Server inplace upgrade – broken network

Windows Server HyperV 2012 > 2012 R2 > 2019

Cannot connect HyperV Manager to HyperV server after in-place upgrade when your domain controller is a virtual machine

So you’ve decided on your upgrade path and committed to action, only to find out your HyperV virtual machines cannot be seen on your network after upgrade.

My home lab operates on 2012 HyperV console, as you should know that means no GUI, which i am very happy with for many reasons. The attack surface is minimal, the resources aren’t needlessly used up and best of all its free

I decided to upgrade, mainly because testing Microsoft Intune meant i needed to enable TPM in HyperV, that and i was due an upgrade anyway

Microsoft HyperV Server 2012 > HyperV 2012 R2 > HyperV 2019

So I downloaded HyperV Server 2012 R2, created a bootable usb and performed an in-place upgrade on my HyperV

Once completed i immediately faced connectivity problems. My home lab is budget, which means my Domain Controller is a VM, yup you guessed it, hosted by the HyperV server

So how do i fix a network when i can’t remotely connect HyperV manager to the HyperV server? The problem being the authentication method is not contactable, the DC as a VM cannot approve the connection request to the HyperV server because the virtual switch is no longer configured properly

My suspicion is that with any in-place windows upgrade, the network adapters always tend to renew in some way shape or form

Know you’re setup

To really understand this problem and the solution I should probably describe my network setup

My hyperV box has 4 VMs and 2 virtual switchs

  • swInternal (Internal LAN switch to my home LAN)
  • swExternal (External WAN switch to my virgin router that is currently in “modem mode”)
  • VM 1 = Firewall gateway (has two virtual NICs swInternal and swExternal)
  • VM 2 = Domain controller (swInternal)
  • VM 3 = webserver (reading this blog from it right now) (swInternal)
  • VM 4 = test machine (constantly being destroyed and re created) (swInternal)

My setup is such that the firewall gateway VM has two network adapters, one for my internal LAN and one for the WAN. The rest of the VMs only have the internal LAN

Identify the problem

  • We cannot connect or see any of the VMs on the LAN, cant ping anything
  • We can’t connect HyperV manager to the HyperV server
  • We can however connect directly using a spare keyboard, mouse and display. Login and then break out PowerShell

Solving the problem

There is a 99% chance that fixing the virtual switches will solve all our problems, so whats the process?

You will need to know a local admin account for your hyperv server

  • 1. Shutdown all VMs
  • 2. Review the current setup. Enumerate the Virtual Switchs, and VM adapters
  • 3. Detach the adapters from the VMs
  • 4. Destroy and recreate the virtual switches
  • 5. Re attach the adapters to the VMs

sounds simple right? ok lets crack on

login to HyperV and execute the following PowerShell commands

This will give you an idea of the current switch setup

From here you can tell I have two hardware NICs and one Virtual NIC on the HyperV machine itself
I also have two virtual switches
And the virtual machines have all been assigned accordingly
In a broken environment you would likely see no IP addresses, moreover the Status’s would be different

So now we need to destroy and recreate everything

  1. Stop all the VM’s
  2. Disconnect the virtual adapters from the VM’s
  3. Remove the virtual switch’s
  4. Recreate the virtual switches
  5. Re attach the virtual adapters

Stop all VM’s

I’ve filtered this command to only stop VM’s that are running

Disconnect the virtual adapters

I’ve filtered this command to filter out the swExternal virtual switch.

Remove the HyperV virtual switch

I only want to remove the swInternal Switch

At this point I decided to rename my Network adapters on the Host, you dont have to do this at this stage.

Re-Create HyperV Virtual Switch

I’ve shown here commands to create the external switches as if the External switch didnt exist. The only difference between the two is that I’m not allowing the swExternal switch management os connectivity.

The difference between a switch category of “internal” & “external” is that internal will isolate the switch from your LAN, keeping internal to the VM host. So basically if you want connectivity to your LAN, always create a virtual switch categorised as External, which is in fact the default category, so you dont need to specify it on creation

If you want to tweak the settings on your newly created switches you can you the following command

Re-Attach the HyperV Virtual machine adapters

I’ve filtered this command to avoid adapters already connected to the swExternal switch here,

Now that the HyperV Host adapters, and Virtual switch’s have been reconfigured you are now ready to start the VMs

At this point i’d recommend restarting your HyperV host, If you have a Domain controller as a VM you’ll need to.
If you are still having connectivity issues you can further diagnose the it by dropping the firewall of the HyperV host by running the following command

If you are having problems with network adapters not setting the correct connection profile, you can force the profile to change

Mopping up

Finally we need to tidy-up the windows.old folder created on the root.
3 steps.
1. Take ownership of the folder
2. reset the security permissions
3. delete the folder

Windows time out of sync (Hyper V)

Time server working but time is wrong

If you have ever come across a situation where your Domain Controller (DC) is syncing properly to its ntp source, but the time is incorrect, the likelihood is that you are operating a virtual server environment and your Primary Domain Controller (PDC) is a virtual machine.

The issue

This Microsoft article Virtual Active Directory Domain Controller explains clearly, uner the heading “Time Service” that you must remove the sync between the virtual machine and the host.

I was late for a gym session because of this

Time service
——————————————————————————–
For virtual machines that are configured as domain controllers, it is recommended that you disable time synchronization between the host system and guest operating system acting as a domain controller. This enables your guest domain controller to synchronize time from the domain hierarchy.

To disable the Hyper-V time synchronization provider, shut down the VM and clear the Time synchronization check box under Integration Services

Thinking about it, this makes perfect sense. With the option ticked, the PDC is set to get its time from the host, and if the host is part of the domain, it in turn is set to get its time from the nearest DC, who also in turn get their time from the PDC. A cyclic loop where no one gets the time from an external ntp source.

To configure the time server please visit here https://blogs.technet.microsoft.com/nepapfe/2013/03/01/its-simple-time-configuration-in-active-directory/

Un-tick the “Time synchrnoization” option from hyper v manager
hyperv-time-synchronization

Windows 8 to Windows 8.1 with redirected user profile folders

You’re here probably because you received this error

Sorry, it looks like this PC can’t run Windows 8.1. This might be because the Users or Program Files folder is being redirected to another partition

The Solution in a nutshell

  • Edit registry keys to re point to system drive
  • Add New Admin account to system
  • Login as new Admin account
  • edit user registry keys to re point to local system users folder
  • Run batch file to create junctions between local system user folders and redirected user folder
  • Restart, login as admin and run windows update to 8.1
  • Once complete edit registry to re point chosen user profile folders back across to redirected drive
  • The user folders that are redirected again must have the AppData hidden folder folder copied across to the redirected user folder fro the system drive

The solution in more detail
Basically we need to trick the updater that the profiles are local. We do this in a few steps

  1. Edit some registry keys
  2. Create a new admin account, restart and login to it
  3. Edit some more registry keys
  4. Create some junctions to point from local profiles to actual profiles
  5. Restart login again as the admin account and run updates

So let’s get started
You might want to save this page as a “.mht” file to the root of your C: so you can refer back to this as you progress
Edit registry
Run regedit as admin and goto
HKLM\SOFTWARE\Microsot\Windows NT\CurrentVersion\ProfileList
Mine looks like this
ProfileList HKLM
Edit the ProfilesDirectory Key back to “%SystemDrive%\Users”
ProfilesDirectory HKLM
Create a new account and mark is as administrator
Restart and login with the new account
Go back into an admin regedit and adjust the remaining keys so they look like this
RESET ProfileList keys HKLM

As you can see I have lots of local logins (I develop apps and as such lots of services login to my machine) so I built a batch file to create the junctions from local to actual user profile folders
For users with no gaps in their name
mklink /J C:\Users\Andre E:\Users\Andre
for users with gaps
mklink /J “C:\Users\.NET v4.5” “E:\Users\.NET v4.5”
My Batch file ended up here c:\MKLinks.bat and looked like this
mklink /J C:\Users\Mcx1-HOME-PC E:\Users\Mcx1-HOME-PC
mklink /J C:\Users\UpdatusUser E:\Users\UpdatusUser
mklink /J C:\Users\Andre E:\Users\Andre
mklink /J “C:\Users\Classic .NET AppPool” “E:\Users\Classic .NET AppPool”
mklink /J “C:\Users\.NET v4.5” “E:\Users\.NET v4.5”
mklink /J “C:\Users\.NET v2.0” “E:\Users\.NET v2.0”
mklink /J “C:\Users\.NET v4.5 Classic” “E:\Users\.NET v4.5 Classic”
mklink /J “C:\Users\.NET v2.0 Classic” “E:\Users\.NET v2.0 Classic”
mklink /J C:\Users\Default E:\Users\Default
mklink /J C:\Users\Public E:\Users\Public

Now we are ready to execute the batch file, so open an elevated command prompt, navigate to the location you saved the batch file and run it.

Next we must adjust the “ProfileImagePath” string value in each user profile sub key of ProfileList (the keys typically start with S-1-5-xx-xxxxxxxxxx-xxxxxxxxxx-xxxxxxxxxx-xxxx) so they point back to the local system drive (C:) like this one

Return Profiles back to system drive locations

Once you are sure you have completed these instructions restart once more, login as the admin account and run windows update.

Once completed, every profile will be put back onto the C:\Users and your folder junctions will have been replaced by these user folders.
There is a simple solution to bring things back across to your redirected drive, editing the users registry key and copying the hidden AppData folder from the C:\Users\Username folder to the target user profile folder on the redirected drive you are trying to use.

I ended up leaving all of my profiles on C and only redirected my profile, which incidentally was far too large to reside on the C:

Exchange Web Service (EWS) configuration

Exchange Web Service (EWS) configuration

The idea here is to create two groups within the Active Directory. The first group will contain the Mailbox accounts you wish to allow access and manipulation of objects within the mailbox (sgEWSImpersonatable). The second will contain the accounts you wish to allow access to the accounts within the first group (sgEWSImpersonate).

What we want to do is

  • AD – Create a security group (sgEWSImpersonateAble), this group will hold the accounts we want to be able to impersonate (eg testAccounts, devsystems etc etc)
  • AD – Create a security group (sgEWSImpersonate), this group will hold the accounts we want to allow impersonation of the accounts in the group sgEWSImpersonateAble
  • EX – Create a Scope (scopeEWSImpersonate), this scope we use to link the ApplicationImpersonation Exchange role to the security group created in the previous step. . Ie we assign the scope to the security group sgEWSImpersonateAble
  • EX – Create a RoleAssignment (mraEWSImpersonation) this Management Role Assignment will be used to tie the ApplicationImpersonation role to the scope. this then compeltes the loop between AD and Exchange

Follow these steps

  1. Create the Security Group in AD (it can be mail enable or not, it makes no difference)
    Group Name: sgEWSImpersonateAble
    Group Description: Exchange Web Service Impersonation, accounts in this group will grant members of the group sgEWSImpersonate impersonation ability via Exchange Web Service calls
    Group Members: TestAccounts, testsqlmailuser, etc,etc,etc
  2. Create the Security Group in AD (it can be mail enable or not, it makes no difference)
    Group Name: sgEWSImpersonate
    Group Description: Exchange Web Service Impersonation, accounts in this group be able to impersonate members of the group sgEWSImpersonateAble via Exchange Web Service (EWS) calls
    Group Members: Developer1,Developer2, Sysadmin1, svcAccount, etc,etc
  3. Create the Scope (This is a one time only requirement to run) In Exchange Management Powershell console run the following, this will link the scope to the groupGet the location of the security group we created for the accounts to impersonate

    >$sgEWSImpersonateAble = $(Get-DistributionGroup sgEWSImperonateAble).Identity.DistinguishedName

    verify we have it by looking at the vaariable
    >$sgEWSImpersonateAble
    CN=sgEWSImpersonateAble,OU=OrganisationalUnitContainingTheGroup,DC=DomainName,DC=local

    Now Create the Scope linking it to the group
    >New-ManagementScope -Name:scopeEWSImpersonate -RecipientRestrictionFilter:”MemberOfGroup -eq ‘$sgEWSImpersonateAble'”

  4. Create the Role Assignment (to link the scope to the group containing the accounts we want to allow impersonation to)>New-ManagementRoleAssignment –Name:mraEWSImpersonation –Role:ApplicationImpersonation –SecurityGroup “sgEWSImpersonate” –CustomRecipientWriteScope: scopeEWSImpersonate

For a long story short execute the following in the Exchange Management Powershell console. Replace the names to those you would prefer.

>$sgEWSImpersonateAble = $(Get-DistributionGroup sgEWSImperonateAble).Identity.DistinguishedName
>$sgEWSImpersonateAble
CN=sgEWSImpersonateAble,OU=OrganisationUnitContainingTheGroup,DC=Domain,DC=local
>New-ManagementScope -Name:scopeEWSImpersonate -RecipientRestrictionFilter:”MemberOfGroup -eq ‘$sgEWSImpersonateAble'”
>New-ManagementRoleAssignment –Name:mraEWSImpersonation –Role:ApplicationImpersonation –SecurityGroup “sgEWSImpersonate” –CustomRecipientWriteScope: scopeEWSImpersonate

Now you can add and remove people and mailboxes to and from the two groups to allow impersonation of mailboxes from accounts

References

http://msdn.microsoft.com/en-us/library/exchange/bb204095(v=exchg.140).aspx

Set-ManagementRoleAssignment
http://technet.microsoft.com/en-us/library/dd335173(v=exchg.141).aspx
New-ManagementRoleAssignment
http://technet.microsoft.com/en-us/library/dd335193(v=exchg.141).aspx
New-ManagementScope
http://technet.microsoft.com/en-us/library/dd335137(v=exchg.141).aspx
Set-ManagementScope
http://technet.microsoft.com/en-us/library/dd297996.aspx

SQL Server Monitor Changes to SCHEMA

If you need to prevent changes to your SQL Server Schema, or at least monitor them, then this post is worth reading

I needed to monitor/control changes made to SQL server databases at server instance level. There are plenty of ways of doing this but there is nothing as simple as a DDL level trigger. It’s clean, can be disabled quickly for rapid change implementation and its a central location which means deploying to sever SQL Servers or a server farm is that much easier.

So the basic requirements
1. We must be able to monitor CREATE,DROP and ALTER on Tables, Views, Stored Procedures, Triggers and Functions
2. We must be able to filter based on user, applying the rule or not
3. We must have a log of each schema change regardless of success or failure
4. We need to send an email when a change has been attempted and was disallowed

So how are we going to do this. Well we can use the ROLLBACK inside our trigger to prevent the changes from occurring, and we can use the RAISERROR to display the appropriate message we want to user/program to receive when the attempt fails. We can also take advantage of the code SQL Server query execution sequence such that after a rollback has been called in a trigger, the parent transaction has been dumped, but any further transactions will continue to be executed and committed. See here http://msdn.microsoft.com/en-us/library/ms187844(v=sql.105).aspx

Some Key fundamentals we are going to be using
First off the

, this will allow us to create the trigger at server level, so it affects all databases. It will hence be located in the Server Objects >> Triggers within SQL Server Management Studio
Next comes

which contains all the details we want about what is changing and who is trying to change it
We will be using “

” to identify the current logged in connection that is behind the event and also filter on this to make a decision to reject or approve the change.
Finally we use the

and

commands to initiate a rejection of the event, followed with our logging insert and in my case an email send method, I won’t get into the details of the mail sending in this article, however its worth mentioning that I have a fully fledged SSIS package driven mail sending engine integrated with MS Exchange, its possible to send from any mailbox to any mail group or address in the active directory or individual mail address book, you can attach files from the network or from binary within the database, you can send with importance in text format or html as this example demonstrates, Its a very powerful and very handy engine to have in the systems topology, took 3 days to write and about a month to fine tune, suddenly sending mail is very easy from anywhere in your infrastructure. Anyway I digress

Enough background, here’s the Code. You will obviously need to adjust bits to suite your needs before you use this
This should be executed against the master

unable to download web platform product list

Unable to download the Web Platform product list from . Check your network connection and try again. If the problem persists, report the issue on the Web Platform Installer forum at: http://go.microsoft.com/fwlink/?LinkId=145244.

Applies to Web platform installer 3 on server 2008

Verify you can browse to http://www.microsoft.com/web/webpi/3.0/WebProductList.xml

If you can then add the following registry key

HKLM\SOFTWARE|Microsoft\WebPlatformInstaller

Add string value (reg_sz) named “ProductXmlLocation”

Value http://www.microsoft.com/web/webpi/3.0/WebProductList.xml

Re run the Web Platform Installer and all should be well

Exchange 2010 and exchange 2003 there is currently no route to the mailbox database

As an Administrator, If during your Exchange 2010 install, when you migrated that single test mailbox from the old Exchange 2003 server into the nice shiny new Exchange 2010 mail server on that new MS Server 2008 64bit you have up and running. You found you couldn’t send mail internally or receive mail internally or in fact receive mail from an external source either, your not alone.

Reviewing the “Queue Viewer” on your Exchange 2010 bi you see there are mails in the queue trying to send with the following error

there is currently no route to the mailbox database

And you also see mails in the inbound queue on your exchange 2003 box.

Fear not, for there is a simple fix. You must Create A Routing Group Connector Between Exchange 2003 and Exchange 2010

According to Microsoft when you installed the new Exchange server 2010 despite the compatibility that exchange 2003 and exchange 2010 can coexist on your domain, they don’t quite let you know that the routing may not properly be configured, my guess is because there are too many permutations of network configurations you might have. So if your like me, you have a single Exchange 2003 server that you want to talk to your Exchange 2010 server then the solution is pretty simple.

Make sure you login to the exchange 2010 box with an account that has “GOD” privileges on your domain
1. Click start
2. in the search box type “Shell”
3. Right click and run “Exchange Management Shell”
4. Copy and paste the following line into notepad

New-RoutingGroupConnector -Name “Interop RGC” -SourceTransportServers “exchange2010FQDN” -TargetTransportServers “Exchange2003FQDN” -Cost 10 -Bidirectional $true -PublicFolderReferralsEnabled $true

Adjust the names accordingly and be sure the quotes are quotes and not some funky character that looks like quotes as is what sometimes happens when copying from the internet.
5. Paste the line into the Exchange Management Shell (EMS) and you should see something not too different from this.

response from adding route, and verifying route exists

6. Now you can verify the creation by running “Get-RoutingGroupConnector”
7. On your exchange 2003 box restart “Simple Mail Transport Protocol” Service
8. On your Exchange 2010 box restart “Mail Exchange Transport” Service

Hope this helped you all

References I used
Message Rerouting and the Unreachable Queue
http://technet.microsoft.com/en-us/library/bb232161.aspx

Routing group connector between an Exchange 2010 organization and Exchange 2003 organization doesn’t exist
A routing group connector between the Exchange 2010 routing group and Exchange 2003 routing groups hasn’t been configured, or the last routing group connector between the Exchange 2010 routing group and Exchange 2003 routing groups has been removed. No routing group connector exists to provide a routing path to the Exchange 2003 recipients. To resolve this problem, first verify that the routing group connector is missing. If that’s the case, you can create a routing group connector. For more information, see Create Additional Routing Group Connectors from Exchange 2010 to Exchange 2003. If a routing group connector does exist, the message is in the Unreachable queue for some other reason. Check the configuration of the routing group connector

Create Additional Routing Group Connectors from Exchange 2010 to Exchange 2003
http://technet.microsoft.com/en-us/library/aa997292.aspx

New-RoutingGroupConnector -Name “Interop RGC” -SourceTransportServers “Ex2010Hub1.contoso.com” -TargetTransportServers “Ex2003BH1.contoso.com” -Cost 10 -Bidirectional $true -PublicFolderReferralsEnabled $true

Upgrade from Exchange 2003 Transport
http://technet.microsoft.com/en-us/library/dd638103.aspx
Exchange Management Shell in Exchange 2010
http://technet.microsoft.com/en-us/library/dd795097.aspx