Category: PowerShell

Tail Windows Update Log Server 2016+ / Windows 10

willjessiamLeave a comment

Sometimes you have to patch mission critical Windows servers and you want to see what’s going on in the background. The modern windows update dialogue provides little verbosity, especially when compared to apt or yum in the Linux world.

In the Windows 7 / Server 2012 days I used to just tail C:\Windows\WindowsUpdate.log and watch the update process, especially when troubleshooting, or when updates were hanging on install. Nowadays if you open that file, you get the following message:

Windows Update logs are now generated using ETW (Event Tracing for Windows).
Please run the Get-WindowsUpdateLog PowerShell command to convert ETW traces into a readable WindowsUpdate.log.

For more information, please visit https://go.microsoft.com/fwlink/?LinkId=518345

As we can see, Microsoft have ditched text based logging in favour of ETW based logging. The link provided takes you to an article on Windows Update log files which lists out the various log files and what they do. If you want a concise log that is the same as what you got in Server 2012, then you have to run Get-WindowsUpdateLog which will spit out that familiar log file. This is no good in my situation, as that generated log file would be for the point in time it was generated, not a dynamically updated log file being written to during updates.

Since I don’t care really about the log contents and I just want to see some verbosity in the update process while its happening, I went through the other logs listed and came across the CBS log. CBS stands for Component based servicing and it provides output on update installation…. great!

When reviewing the log, I identified that the lines containing the strings Appl and FOD make up the majority of the hundreds, or thousands of entries, when running a tail. I filtered these out and it seems that what’s left is mostly operational logging to do with the update engine. Since my goal is to watch Windows Update progress to ensure something is happening, I wrote a quick PowerShell one liner:

Get-Content -Path $env:WinDir\Logs\CBS\CBS.log -Tail 10 -Wait | Select-String -Pattern 'Appl:|FOD:' -NotMatch

This command will get last 10 lines of the CBS log file, wait for new entries to be written, filter out Appl and FOD and write the results to the console.

If I want to get more details after patching is complete, I can always run Get-WindowsUpdateLog.

This may stop working at any point of course if Microsoft change something, but for now, it gives me peace knowing that the update stuck at 23% for 47 minutes is actually being installed and I’m not about to have a long night… or not too long anyway.

Thanks for reading – Jesse

Get Windows 10 Version Information

willjessiamLeave a comment

This morning I needed to quickly get Windows 10 version information for all workstations in a domain, so I wrote the below PowerShell script. This basic script only needed a few things to get the job done; a way to get all workstations in the domain, a way to only check computers that are online at the time, so the script doesn’t take too long with all the failures generated by offline systems, and the registry keys to determine the current OS build and minor version.

Import-Module -Name ActiveDirectory

$all_computers = Get-ADComputer -Filter * -SearchBase 'OU=computers,DC=somedomain,DC=com' | Select-Object -Property Name

$ExportPath = "$env:TEMP\$(Get-date -Format 'yyyyMMddhhmmss')_workstation_os_build_report.csv"

foreach ($c in $all_computers.name)

{

    if (Test-Connection -ComputerName $c -count 1 -Quiet ) {

    Write-Host "Processing $c" -ForegroundColor Cyan

    $CurrentBuild = ""
    $UBR = ""
    $OSVersion = ""
    $ComputerSystem = ""
    $props = ""
    $obj = ""

        $CurrentBuild = Invoke-Command -ComputerName $c -ScriptBlock { (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion' –Name CurrentBuild).CurrentBuild } -ErrorAction SilentlyContinue
        $UBR = Invoke-Command -ComputerName $c -ScriptBlock { (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion' –Name UBR).UBR } -ErrorAction SilentlyContinue
        $OSVersion = $CurrentBuild + "." + $UBR
        $ComputerSystem = Get-WmiObject -ComputerName $c -Class Win32_ComputerSystem -ErrorAction SilentlyContinue

       $props = [ordered]@{ 
        'HostName' = $ComputerSystem.Name;
        'OSVerion' = $OSVersion
        }
        $obj = New-Object -TypeName PSObject -Property $props

    Write-Output $obj | Export-Csv -Path $ExportPath -NoTypeInformation -Append -NoClobber -Force
    }
    else {
        Write-Host "$c is offline..." -ForegroundColor Green
    }

}
Write-Host "Output csv file is located here: `n `n $ExportPath `n" -ForegroundColor Yellow

Obviously there are a few issues with this script. It won’t get systems that are turned off, PowerShell remoting needs to be enabled/working, and in a large domain, it’s probably going to take a long time without some sort of parallelisation or a more efficient way of querying each host…. regardless, this was just a quick indicator for me of the general patch levels of systems in the domain.

Code also on my github here.

Thanks for reading – Jesse

Get all AD group members with PowerShell

willjessiamLeave a comment

I was recently doing an audit of AD group memberships and since I find it easier to do this by filtering a spreadsheet, I needed to get all groups and their members out to a CSV. This basic script does the job and captures key properties like the name, DN and SID for the group as well as the name, DN, SID and object class for the member. This information would be enough to re-create a group structure and re-populate members if you needed to. 

# Get All AD Group members for all groups

$groups = Get-ADGroup -Filter *

foreach ($group in $groups) {

$members = Get-ADGroupMember -Identity $group

    foreach ($member in $members) {

            [PSCustomObject]@{
            GroupName = $group.Name
            GroupDN = $group.DistinguishedName
            GroupSID = $group.SID
            MemberName = $member.name
            MemberDN = $member.DistinguishedName
            MemberSID = $member.SID
            MemberObjectClass = $member.ObjectClass
            } | Export-Csv -Path C:\temp\all_adgroupmembers_20220323_1.csv -NoClobber -NoTypeInformation -Append 
        }

}

Code also on my github here.

Thanks for reading – Jesse

Exchange Online message trace with more detail

willjessiamLeave a comment

Recently I needed to dig through some email using the Exchange Online PowerShell module and I found the default cmdlets a bit lacking in detail. Get-MessageTrace and Get-MessageTraceDetail show you enough, but sometimes you want to know more about the flow of an email from when it was received until it was ultimately delivered, marked as spam or quarantined. The graphical view at https://security.microsoft.com/quarantine is good and does give you pretty much everything you need, but I wanted to be able to see the spam scoring metrics and any additional details.

Get-MessageTrace gives us information about when the message was received, the sender and recipient addresses, the subject and what ultimately happened to the email. See the following screen shot where I’ve used a spam email that was captured in quarantine to demonstrate:

Get-MessageTrace -StartDate (get-date).AddDays(-10) -EndDate (get-date) -SenderAddress ds043_buh@edu.klgd.ru

Get-MessageTraceDetail give us more specifics about each event that occurred from when the message was received, to in this case, when it was quarantined.

Get-MessageTrace -StartDate (get-date).AddDays(-10) -EndDate (get-date) -SenderAddress ds043_buh@edu.klgd.ru | Get-MessageTraceDetail

But what happens if you want to know more about each event and scrutinise further? The answer is in the ‘data’ property of each event, which is an xml string. If I store the message trace detail in a variable, select an event and look the data property, I see something like the following:

$messageTraceDetail = Get-MessageTrace -StartDate (get-date).AddDays(-10) -EndDate (get-date) -SenderAddress ds043_buh@edu.klgd.ru | Get-MessageTraceDetail

$messageTraceDetail[0].Data

As we can see, this data is not very useful. To make it more useful, we can create an xml object from this xml string so that we can work with each property and do something with it.

$xml = [xml]$messageTraceDetail[0].Data
$xml.root.MEP

The data we are after lives under the root.MEP node of the xml object. Pathing to this exposes the properties in the event we have selected, which in this example is the receive event.

My goal was to expose the details of each event into a single object and have it output to the console for inspection. To do this, I needed to get the message, get message detail, loop through each event, convert the data property to xml, select the properties that were relevant and build a custom object.

As I started writing the script, I came across a few hurdles that needed addressing:

  • When adding properties to my custom object, some event properties were not showing any values. I realised that some event property values were strings and some integers.
  • Some property names were the same between events, so I needed a way to make these unique if they were going to be members of the same custom object.
  • Some properties legitimately had no data in their values since not all delivered mail is the same. For example, a spam email may populate a spam list property value, but a clean email would not. I needed to remove blank properties dynamically each time the script ran.
  • The [datetime] objects returned in some properties were set to UTC +0 and I wanted to see these properties in local time.
  • The ‘RecipientReference’ property was always blank (in all the emails I’ve checked), so I wanted to exclude this all together.
  • Not all data was available in the Get-MessageTraceDetail events data property, some of it was in the detail property and some was in the output from Get-MessageTrace.
  • I wanted to be prompted to enter a recipient or subject and filter on these if required.

The below script is the result and the output looks something like this:

function Get-MessageTraceWithMoreDetail
{
<#
.Synopsis
   Trace mail messages with more detail
.DESCRIPTION
   This script traces mail messages and provides more detail exposing each event and outputs a single object for review.
   Start date and end date values are set to 10 days old and current date respectively, but you can overide if required.
.EXAMPLE
   Get-MessageTraceWithMoreDetail -startDate (Get-Date).AddDays(-3) -endDate $endDate = (Get-Date).AddDays(-1)
#>
    [CmdletBinding()]
    Param
    (
        [Parameter(Position=0)]
        [datetime]$startDate = (Get-Date).AddDays(-10),

        [Parameter(Position=1)]
        [datetime]$endDate = (Get-Date)
    )

try {

        $senderAddress = Read-Host -Prompt "What is the sender address or domain? eg. @domain.com or user@domain.com"

        Clear-Variable -Name recipientCheck,subjectCheck -ErrorAction SilentlyContinue

        while ($recipientCheck -notmatch "Y|N")
        {
            $recipientCheck = Read-Host -Prompt "Do you want to search using a recipient filter? [ Y | N ]"
        }

        while ($subjectCheck -notmatch "Y|N")
        {
            $subjectCheck = Read-Host -Prompt "Do you want to search using a subject filter? [ Y | N ]"
        }

        if ($subjectCheck -eq "Y" -and $recipientCheck -eq "Y") {
        
                $recipientFilter = Read-Host -Prompt "What is the recipients email address? eg. user@domain.com"
                $subjectFilter = Read-Host -Prompt "What words does the subject contain?"

                $messagesToReview = Get-MessageTrace -StartDate $startDate -EndDate $endDate -SenderAddress $senderAddress -RecipientAddress $recipientFilter | Where-Object -FilterScript {$_.Subject -like "*$subjectFilter*"}

        }
        elseif ($subjectCheck -eq "Y" -and $recipientCheck -eq "N") {

                $subjectFilter = Read-Host -Prompt "What words does the subject contain?"

                $messagesToReview = Get-MessageTrace -StartDate $startDate -EndDate $endDate -SenderAddress $senderAddress | Where-Object -FilterScript {$_.Subject -like "*$subjectFilter*"}

        }
        elseif ($subjectCheck -eq "N" -and $recipientCheck -eq "Y") {

                $recipientFilter = Read-Host -Prompt "What is the recipients email address? eg. user@domain.com"

                $messagesToReview = Get-MessageTrace -StartDate $startDate -EndDate $endDate -SenderAddress $senderAddress -RecipientAddress $recipientFilter

        }
        else {

               $messagesToReview = Get-MessageTrace -StartDate $startDate -EndDate $endDate -SenderAddress $senderAddress

        }

    Write-Host ""
    Write-Host ""
    Write-Host "============ Message Report $(get-date) ============" -ForegroundColor Cyan

    foreach ($message in $messagesToReview) {

                # Convert to local time
                $messageReceivedDate = Get-LocalTime -UTCTime $message.Received


                $customMessageObjectProps = [ordered]@{
                    'Step 0 : Sender IP' = $message.FromIP ;
                    'Step 0 : From address' = $message.SenderAddress ;
                    'Step 0 : Date received' = $messageReceivedDate ;
                    'Step 0 : To address' = $message.RecipientAddress ;
                    'Step 0 : Message subject' = $message.Subject ;
                    'Step 0 : Message status' = $message.Status ;
                    'Step 0 : Message size (KB)' = $([math]::Round(($message.Size / 1KB),2)) ;
                    'Step 0 : Message ID' = $message.MessageId}

                $customMessageObject = ""

                $customMessageObject = New-Object -TypeName PSObject -Property $customMessageObjectProps

                $messageDetail = $message | Get-MessageTraceDetail

                [int]$c = ""

                foreach ($event in $messageDetail) {

                    $xml = [xml]$event.Data
                    $eventReport = $xml.root.MEP

                    $c++
                    
                    # Convert to local time
                    $adjustedDate = Get-LocalTime -UTCTime $event.Date
             
                    $customMessageObject | Add-Member -NotePropertyName "Step $($c) : Action taken:" -NotePropertyValue $($event.Detail)
                    $customMessageObject | Add-Member -NotePropertyName "Step $($c) : Action time:" -NotePropertyValue $($adjustedDate.ToString())
            

                        foreach ($xmlProp in $eventReport) {

                            if ( 'string' -in (($eventReport | Get-Member).Name)) {
                
                                   if ($xmlProp.String -ne $null) {

                                        if ($xmlProp.Name -ne "RecipientReference") {
                        
                                        $customMessageObject | Add-Member -NotePropertyName "Step $($c) : $($xmlProp.Name)" -NotePropertyValue $xmlProp.string -Force

                                        }
                       
                                   }           
                
                            }
                            elseif ( 'integer' -in (($eventReport | Get-Member).Name)) {
                
                                   if ($xmlProp.integer -ne $null) {
                       
                                        $customMessageObject | Add-Member -NotePropertyName "Step $($c) : $($xmlProp.Name)" -NotePropertyValue $xmlProp.integer -Force
                       
                                   }           
                
                            }
                        }
                
                   }

        Write-Output $customMessageObject

        }

    }
    catch {

    Write-Host "Failed to perform message trace with more detail"
    Write-Host "$($_)"
    Write-Host "Line Number: $($_.InvocationInfo.ScriptLineNumber)"
    Write-Host "Offset: $($_.InvocationInfo.OffsetInLine)"
    Write-Host "Line: $($_.InvocationInfo.Line)"

    }

}


function Get-LocalTime($UTCTime)
{
$strCurrentTimeZone = (Get-WmiObject win32_timezone).StandardName
$TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById($strCurrentTimeZone)
$LocalTime = [System.TimeZoneInfo]::ConvertTimeFromUtc($UTCTime, $TZ)
Return $LocalTime
}

Code is also on my GitHub.

Apple iPad Activation via MDM and courier.push.apple.com

willjessiamLeave a comment

Activating an iPad with your MDM platform is usually a straight forward process.Your hardware vendor associates your device enrollment program (DEP) code with the device, your DEP platform tells your MDM that the device belongs to you and when you turn the device on for the first time, it phones home, gets directed to MDM, your configuration profiles are pushed and the iPad gets activated, enrolled and managed. Simple right?

Turns out, there is an odd thing that Apple does that may cause problems if you use a proxy or content gateway for accessing the internet when you do the initial activation.

Enter courier.push.apple.com. This is a placeholder DNS record, which is designed to assist with load balancing the activation process. As we’ll find out, the way this DNS record is used is not standard and it helps to understand how it works when troubleshooting activating a device behind a proxy.

First, some lookups:

PS C:\> Resolve-DnsName -Name courier.push.apple.com -DnsOnly

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
courier.push.apple.com         CNAME  279   Answer     courier-push-apple.com.akadns.net

So courier.push.apple.com is a CNAME for courier-push-apple.com.akadns.net . Let’s see where courier-push-apple.com.akadns.net goes.

PS C:\> Resolve-DnsName -Name courier-push-apple.com.akadns.net -DnsOnly

Name                        Type TTL   Section    PrimaryServer               NameAdministrator           SerialNumber
----                        ---- ---   -------    -------------               -----------------           ------------
akadns.net                  SOA  180   Authority  internal.akadns.net         hostmaster.akamai.com       1560251729

As we can see courier-push-apple.com.akadns.net does not resolve. It just returns the SOA record for the domain.

We can also confirm that this is not a geographical DNS inconsistency by querying alternate DNS servers in different parts of the world.

PS C:\> Resolve-DnsName -Name courier-push-apple.com.akadns.net -DnsOnly -Server 8.8.8.8

Name                        Type TTL   Section    PrimaryServer               NameAdministrator           SerialNumber
----                        ---- ---   -------    -------------               -----------------           ------------
akadns.net                  SOA  163   Authority  internal.akadns.net         hostmaster.akamai.com       1560251729

PS C:\> Resolve-DnsName -Name courier-push-apple.com.akadns.net -DnsOnly -Server 1.1.1.1

Name                        Type TTL   Section    PrimaryServer               NameAdministrator           SerialNumber
----                        ---- ---   -------    -------------               -----------------           ------------
akadns.net                  SOA  133   Authority  internal.akadns.net         hostmaster.akamai.com       1560251729

PS C:\> Resolve-DnsName -Name courier-push-apple.com.akadns.net -DnsOnly -Server 211.11.195.114

Name                        Type TTL   Section    PrimaryServer               NameAdministrator           SerialNumber
----                        ---- ---   -------    -------------               -----------------           ------------
akadns.net                  SOA  180   Authority  internal.akadns.net         hostmaster.akamai.com       1560251729

PS C:\> Resolve-DnsName -Name courier-push-apple.com.akadns.net -DnsOnly -Server 212.234.34.121

Name                        Type TTL   Section    PrimaryServer               NameAdministrator           SerialNumber
----                        ---- ---   -------    -------------               -----------------           ------------
akadns.net                  SOA  145   Authority  internal.akadns.net         hostmaster.akamai.com       1560251729

PS C:\> Resolve-DnsName -Name courier-push-apple.com.akadns.net -DnsOnly -Server 167.233.5.204

Name                        Type TTL   Section    PrimaryServer               NameAdministrator           SerialNumber
----                        ---- ---   -------    -------------               -----------------           ------------
akadns.net                  SOA  14    Authority  internal.akadns.net         hostmaster.akamai.com       1560251729

Google, Cloudflare as well as name servers in France, Germany and Korea all return an SOA for courier-push-apple.com.akadns.net

A quick google of this strange behavior reveals people resolving load balanced endpoints for courier-push-apple.com.akadns.net, namely 'x'.courier-push-apple.com.akadns.net, eg. 1.courier-push-apple.com.akadns.net , 2.courier-push-apple.com.akadns.net , 3.courier-push-apple.com.akadns.net etc.

After working with an extremely helpful engineer at Apple, it was determined that the load balancing for activating iPad’s is done on the device its self. Rather than DNS telling the device which endpoint to hit, the device prepends a number like '1' to the start of courier-push-apple.com.akadns.net to make it 1.courier-push-apple.com.akadns.net. When we look up these names, we get a much different result. 

PS C:\> Resolve-DnsName -Name 1.courier-push-apple.com.akadns.net -DnsOnly

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
1.courier-push-apple.com.akadn CNAME  48    Answer     apac-au-courier-4.push-apple.com.akadns.net
s.net

Name       : apac-au-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 15
Section    : Answer
IP4Address : 17.57.145.37


Name       : apac-au-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 15
Section    : Answer
IP4Address : 17.57.145.36

PS C:\> Resolve-DnsName -Name 2.courier-push-apple.com.akadns.net -DnsOnly

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
2.courier-push-apple.com.akadn CNAME  60    Answer     apac-au-courier-4.push-apple.com.akadns.net
s.net

Name       : apac-au-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 60
Section    : Answer
IP4Address : 17.57.145.36


Name       : apac-au-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 60
Section    : Answer
IP4Address : 17.57.145.37

PS C:\> Resolve-DnsName -Name 3.courier-push-apple.com.akadns.net -DnsOnly

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
3.courier-push-apple.com.akadn CNAME  19    Answer     apac-au-courier-4.push-apple.com.akadns.net
s.net

Name       : apac-au-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 55
Section    : Answer
IP4Address : 17.57.145.36

Name       : apac-au-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 55
Section    : Answer
IP4Address : 17.57.145.37

PS C:\> Resolve-DnsName -Name 10.courier-push-apple.com.akadns.net -DnsOnly

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
10.courier-push-apple.com.akad CNAME  55    Answer     apac-au-courier-4.push-apple.com.akadns.net
ns.net

Name       : apac-au-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 55
Section    : Answer
IP4Address : 17.57.145.37

Name       : apac-au-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 55
Section    : Answer
IP4Address : 17.57.145.36

Here in Australia, the 'x'.courier-push-apple.com.akadns.net endpoints are a CNAME for an A record of apac-au-courier-4.push-apple.com.akadns.net which resolves to the same two load balanced IPs.

If however I query a name server in Germany, i get much different results.

PS C:\> Resolve-DnsName -Name 10.courier-push-apple.com.akadns.net -DnsOnly -Server 167.233.5.204

Name                           Type   TTL   Section    NameHost
----                           ----   ---   -------    --------
10.courier-push-apple.com.akad CNAME  5     Answer     eu-central-courier-4.push-apple.com.akadns.net
ns.net

Name       : eu-central-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 52
Section    : Answer
IP4Address : 17.57.146.165

Name       : eu-central-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 52
Section    : Answer
IP4Address : 17.57.146.169

Name       : eu-central-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 52
Section    : Answer
IP4Address : 17.57.146.164

Name       : eu-central-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 52
Section    : Answer
IP4Address : 17.57.146.167

Name       : eu-central-courier-4.push-apple.com.akadns.net
QueryType  : A
TTL        : 52
Section    : Answer
IP4Address : 17.57.146.166

This time we get 5 load balanced IPs in the Central Europe region. None of this is unusual of course and we expect this type of behavior from modern CDN’s. The slightly unusual thing here is that courier-push-apple.com.akadns.net resolves to nothing and this is by design. What is supposed to happen during device activation is that the device queries courier.push.apple.com, which returns courier-push-apple.com.akadns.net. The very act of receiving this name host as a response on the device tells the code that is running during activation to prepend a number to courier-push-apple.com.akadns.net, making it a valid DNS query. The device now hits 'x'.courier-push-apple.com.akadns.net and receives a response from a server which allows the normal activation process to occur.

Now to the point of the article. What happens if your device requests courier.push.apple.com via your proxy server and the proxy looks up courier.push.apple.com to make sure its valid before returning the response. The proxy sees that the resultant response of courier-push-apple.com.akadns.net resolves to nothing and squashes the response since it goes nowhere. The device now, sitting and waiting for a response, never gets it and decides that it wont proceed with activation, because no server is responding to it.

How do you fix this issue? Depends on your proxy. Depends on your environment. I don’t currently have an answer but I’m working on it and may do another blog post depending on the outcome. Hopefully if your stuck on this issue this clarifies things for you in regards to how iPads interact with courier.push.apple.com.

Side note: This article is helpful to review when looking at push notification firewall requirements.

Thanks for reading – Jesse

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 Cipher with Server 2012 R2

willjessiamLeave a comment

I was recently working on an issue where a monitoring system that runs on top of Server 2012 R2 was not able to establish a TLS handshake with web servers to check their availability. After some investigation and a ticket with Microsoft, it was determined that SCHANNEL on Server 2012 R2 does not support modern ciphers (a few posts on Stack Overflow confirms the same thing).

I find it crazy that Microsoft don’t support modern ciphers on server operating systems that are still in support (albeit extended support in this case). Surely as web servers start to upgrade their cipher suites, this is going to break stuff and everyone will be forced to upgrade to newer Server OS’s whether they are ready to or not.

The troubleshooting process with Microsoft involved the following steps:

Make sure that the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 cipher exists in the following registry locations:

KeyValueType
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010002FunctionsREG_MULTI_SZ
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002FunctionsREG_SZ

As well as this, the Microsoft support engineer advised to enable strong crypto in dot net framework by configuring the following key:

KeyValueType
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft.NETFramework\v4.0.30319SystemDefaultTlsVersionREG_DWORD

Lastly, to make sure that a TLS 1.2 connection was established, the following registry keys needed to be set to enforce TLS 1.2 and disable older SSL and TLS protocols.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\Multi-Protocol Unified Hello]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\Multi-Protocol Unified Hello\Client]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\Multi-Protocol Unified Hello\Server]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Client]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Server]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Client]
"DisabledByDefault"=dword:00000001
"Enabled"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Client]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

Once these registry keys were set and after a reboot, we could do some testing. The quickest way to determine what was happening was to run a packet capture and use Internet Explorer to browse to the site that was attempting to handshake using the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 cipher. Internet Explorer uses SCHANNEL for establishing TLS connections, so this is an easier test than mucking about with your app that also uses SCHANNEL. For the capture, I wrote a quick PowerShell script to use the built in netsh tool to capture packets and convert them to .pcapng for easy analysis using Wireshark.

# NetworkCapture.ps1

function startCapture {
    param ()
netsh trace start capture=yes report=disabled
}

function stopCapture {
    param ()
netsh trace stop
}

function convertCapture {
    param ()
# Setup function variables
$ETLToolZipPath = "$env:USERPROFILE\Downloads\etl2pcapng.zip"
$CapturePath = "$env:LOCALAPPDATA\Temp\NetTraces\NetTrace.etl"
$PCAPNGPath = "$env:LOCALAPPDATA\Temp\NetTraces\$(Get-Date -Format dd-MM-yyy-hhmm)_NetTrace.pcapng"
$ETLToolPath = "$env:USERPROFILE\Downloads\etl2pcapng\x64\"    

if ($ETLToolPath) {
    Write-Host "Tool already exists, skipping..."
}
else {
# Get Windows ETL to PCAPNG tool
Invoke-WebRequest -Uri https://github.com/microsoft/etl2pcapng/releases/download/v1.4.0/etl2pcapng.zip -OutFile $ETLToolZipPath -UseBasicParsing 

# Unzip the archive
Expand-Archive -Path $ETLToolZipPath -DestinationPath $env:USERPROFILE\Downloads\ -Force
}

# Run conversion
Set-Location -Path $ETLToolPath
.\etl2pcapng.exe $CapturePath $PCAPNGPath

Write-Output -InputObject "Capture has been converted and can be found here: $PCAPNGPath"
}

Here is the result from running the three functions listed in the above script. In between the startCapture and stopCapture functions, using Internet Explorer, I browsed to the site using the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 cipher.

PS C:\> startCapture
Trace configuration:
-------------------------------------------------------------------
Status:             Running
Trace File:         C:\Users\lab\AppData\Local\Temp\NetTraces\NetTrace.etl
Append:             Off
Circular:           On
Max Size:           250 MB
Report:             Disabled

PS C:\> stopCapture
Merging traces ... done
File location = C:\Users\lab\AppData\Local\Temp\NetTraces\NetTrace.etl
Tracing session was successfully stopped.

PS C:\> convertCapture
Tool already exists, skipping...
IF: medium=eth  ID=0    IfIndex=11
IF: medium=eth  ID=1    IfIndex=14
Converted 208 frames
Capture has been converted and can be found here: C:\Users\lab\AppData\Local\Temp\NetTraces\01-02-2021-0856_NetTrace.pcapng

The results of the trace confirmed that the TLS 1.2 hello was not successfully negotiated:

From here we see the client reaches out with the TLSv1.2 hello, but further down the server returns with “Handshake Failure”.

In the Windows system event log, we also see the following SCHANNEL error:

At this point I did some additional testing using newer operating systems. I already knew that the sites in question worked using Windows 10 but I wanted to confirm with Server 2016 and Sever 2019. As expected, SCHANNEL supports the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 cipher in newer versions of Windows and Internet Explorer is able to connect. The odd thing with Internet Explorer is that it throws the following error on Server 2012 R2, which is a bit misleading as RC4 is not in use or enabled:

This page can’t be displayed

Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to https://site.example.com again. If this error persists, it is possible that this site uses an unsupported protocol or cipher suite such as RC4 (link for the details), which is not considered secure. Please contact your site administrator.

Another thing that proves the issue is with SCHANNEL is that Chromium on Server 2012 R2 works and you can access TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 negotiated sites fine. This is because Chromium does not use SCHANNEL for establishing TLS.

Ultimately, after collecting some more logs and a bit of back and forward on email, Microsoft came back stating that TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 is not supported on Windows Server 2012 R2 and that newer ciphers can be considered a windows ‘feature’ that has been introduced on newer operating systems only. They also provided this article as a reference.

If you come across this issue and your app uses SCHANNEL to establish TLS connections, you may be in for an OS upgrade.

Thanks for reading. -Jesse