RE: Turning off NTLMv1 on the NETID domain controllers

July 1, 2014

This is an update for the high-impact service change in 2 weeks.

 

More Info:

  • NTLMv1 use is significantly down in the server log files available to the UWWI service team. Last week’s logs suggest that as many as 75% of the misconfigured computers we were seeing a month ago have now been fixed. The UWWI service does not have access to your log files. Only you can check whether your users will be affected. See the original announcement below for resources to help you do that.
  • An update on the non-Windows browser known problem we mentioned last week: Making a change to an IIS web server which is configured to use Windows Integrated authentication may be a workaround to consider. We’ve added removing Integrated Windows authentication and adding Basic authentication (with SSL required) as a workaround to our documentation. The Dynamics AX service plans to apply this workaround. If you do have a IIS web server with Windows Integrated enabled, you should check your logs for NTLMv1 use.
  • We continue to email user notifications to those users that are in the log files we have access to. We sent a round of notifications today to 250 users. We plan to send additional user notifications on: 7/8, 7/14, and 7/15.

 

 

From: Brian Arkills Sent: Tuesday, June 24, 2014 2:16 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for the high-impact service change in 3 weeks.

 

More Info:

  • We’ve updated the known problems/workarounds documentation with what we think is a substantial addition. For non-Windows clients interacting with a web server leveraging Windows Integrated authentication, we are aware of only one browser that supports NTLMv2: Safari on MacOS. It’s possible there are other options, but we are unaware of them. Aside from using Safari, an alternative workaround for a non-Windows client would be to get Kerberos authentication configured on that client. A possible workaround on the web server side is to remove “NTLM” (i.e. Windows Integrated) and leave “Negotiate” (and require https)—and consider using federated authentication protocols in the future.
  • The 1st round of user notifications based on log entries available to UWWI happened on 6/16. That set of user notifications came from log entries on Exchange, Sharepoint, and NETID domain controller servers. More rounds of user notifications are planned, and we will add Dynamics AX server logs as a source. If you have log entries you’d like included in our user notification process, please let us know. We’d be happy to walk you through using the PowerShell script we previously made available to everyone, if you need assistance.
  • We plan to add another resource based on feedback. There will be a web application that only accepts NTLMv2 to allow clients to verify their computers are configured correctly. More info when that resource is available.

 

From: Brian Arkills Sent: Friday, June 6, 2014 11:54 AM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: RE: Turning off NTLMv1 on the NETID domain controllers

 

This is an update for this high-impact service change.

 

A date for the change has been set:

 

What and When:

On July 16 at 10am we plan to turn off NTLMv1 support on the NETID domain controllers.

 

More Info:

Because NTLMv1 use persists in large numbers, over the next couple weeks we will be directly contacting users which our logs show still are using NTLMv1. If you are their local IT support, they may contact you for assistance as a result of these notifications. A sample notification email is attached.

 

We strongly encourage IT staff to proactively identify and correctly configure computers they support to not use NTLMv1 before July 16. See the prior announcement below for the methods we’ve developed to help you do that.

 

We’ve also updated the logon data we previously published to include two more sets of log data we’ve collected & analyzed since the prior announcement. But as noted previously, our log data will not cover all cases, so you should not rely solely on it. See https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons for all the log data we’ve collected, as well as the list of users we currently plan to directly notify. Our list may grow or shrink based on subsequent log data.

 

From: Brian Arkills Sent: Tuesday, April 22, 2014 12:58 PM To: ‘uwwi-announce@uw.edu’ (uwwi-announce@uw.edu) Subject: Turning off NTLMv1 on the NETID domain controllers

 

A high-impact service change is planned for the UWWI NETID domain service. This notification will be sent to a variety of mailing lists to broadly increase awareness.

 

What and When:

This summer we plan to turn off NTLMv1 support on the NETID domain controllers. We have not yet set a date for this change because of the amount of proactive mitigation still needed. Later in Spring quarter, we expect to set a specific date for the summer.

 

Why:

A greatly increased threat profile from cloud-based NTLMv1 cracking tools has emerged over the past year, growing pressures due to UW identity assurance initiatives, and the passing of Windows XP mean it is time for NTLMv1 to be retired.

 

What you need to do:

On 8/1/2013, we made this change and rolled it back because of widespread impact. We don’t plan to roll back this change, so you should prepare for this change ahead of time. The cause of problems is primarily in your hands—workstations and member servers with a poorly configured LMCompatibilityLevel setting.

 

The good news is that we’ve done a lot of work to help assist you in getting things fixed up.

 

There are several things we’d like IT support staff to do:

  1. Adjust any group policies that are setting the LMCompatibilityLevel to eliminate NTLMv1 in your domain. The group policy setting is: “Computer/Policies/Windows Settings/Local Policies/Security Options/Network Security: LAN Manager authentication level”. IT staff can see our LMCompatibilityLevel Guidance document for how to proceed.
  2. Download the PowerShell script we created. Use it to query your domain controller’s security logs for NTLMv1 logon events. Apply the documented workarounds to the computers that come up in those events. Next use it to query important member server’s security logs for NTLMv1 logon events. Again, apply workarounds. Repeat this process a couple times over the months ahead until you are comfortable you didn’t miss anyone. Don’t assume that just querying your domain controllers will unearth all the problems—that’s the mistake we made preparing 8 months ago. J
  3. Read the documentation of known problems and workarounds. Also read the customer document we’ve prepared to help those users that don’t have someone to help them—feel free to re-use it. Be ready to use this documentation to troubleshoot and apply the appropriate workaround on the date of the change.
  4. Check the details of UW-IT’s analysis of its logs for a short period of time (i.e. this is not a comprehensive list of everything that needs attention). We have a simplified list of the raw NTLMv1 logon events (a timestamp, UW NetID, hostname triad), along with a list of the unique UW NetIDs and unique hosts across all those logon events. Go here to see an excel spreadsheet with the list. You probably support one or more of these computers/users involved and can proactive fix these. We plan to direct contact anyone we know is still using NTLMv1 in a month’s time, and the users you support likely will call on you at that time if you don’t proactively help them. We’d encourage you to look at our list and get what you can fixed now.Resources:
  5.  

Known NTLMv1 Logons: https://wiki.cac.washington.edu/display/UWWI/Known+NTLMv1+Logons

Known problems and workarounds: https://wiki.cac.washington.edu/display/UWWI/NTLMv1+Removal+-+Known+Problems+and+Workarounds

PowerShell script to identify misconfigured computers: https://wiki.cac.washington.edu/display/UWWI/Using+Get-NtlmV1LogonEvents.ps1

PowerShell script to correctly set the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/Using+Set-LMCompatibilityLevel.ps1

IT focused guidance on how to approach changing the LMCompatibilityLevel: https://wiki.cac.washington.edu/display/UWWI/LMCompatibilityLevel+Guidance

Customer focused guidance on how to fix NTLMv1 on their computer: https://wiki.cac.washington.edu/pages/viewpage.action?pageId=64035299

Uwwi-discuss mailing list–to join: http://mailman.u.washington.edu/mailman/listinfo/uwwi-discuss

 

If you have questions about this planned work, would like some consultation or assistance in proactively preparing, or would like to report a problem or workaround not in the known problem documentation, please send email to help@uw.edu with “UWWI NTLMv1 DC work” in the subject line. We’d love to help folks eradicate NTLMv1, so don’t be shy. J

 

Brian Arkills

UW-IT, Identity and Access Management

UWWI Service Manager