The U Drive suffered an interruption of service on June 7th 2017. The root cause looks to be that the U Drive was not able to handle the very high load that was created by customer computers behaving in a strange way. We are still working to identify exactly why these systems are requesting thousands of mounts of the U Drive, but the risk has been mitigated with rate limits placed on how many times any individual machine may mount the U Drive in one minute. The current rate is set to 60 new mounts per IP address, per minute. Only a very small number of machines are attempting any more mounts than that. The U Drive is considered stable in this configuration.
We have seen a few cases of a new issue, however, and I am hoping the larger tech support community can help gather information and maybe find an answer.
We have had 6 systems reported as having very slow opening and saving performance for Word documents (taking roughly a minute for either action). Non Microsoft Office documents on these same systems are not slow. Testing has shown that other systems ‘near’ the affected systems do not show this problem. All the affected systems have been Windows 7 desktop systems, using different versions of Office (2010, 2013, 2016). Other Windows 7 desktop systems using the same versions of Office have been shown to not have the issue. So far only Word documents have been seen to have this issue, but other Office documents might have this problem as well.
Users that have created new accounts on their workstations, or upgraded to Windows 10 (which modifies the local account a great deal) report that this resolves this issue.
We would like to gather more data by asking that people fill out the following short survey about if they are seeing the problem or not. That may help isolate some variable, and will also give us a list of people to inform when a solution is found, or when we have additional information to share. The questionnaire asks if you are seeing slow performance with MS Word on the U Drive, the version of Office you have, if you are seeing normal performance for non-Office files, your OS version, and the brand of computer you are on. I am interested in hearing back both from people who are NOT seeing the issue as well as those that are. The survey can be found at:
https://goo.gl/forms/F78IrP1TFq8rEOEq2
If you are experiencing this issue, working directly with local support staff that have experience with your system or similar configurations may be very helpful in finding a solution.
I am also asking that the tech staff working directly with systems suffering from this issue share any information you have by emailing help@uw.edu with the subject line “U Drive slowness with MS Word”. My hope is that your expertise with OS and application support will find the Office Preference setting that needs tweaking, or the registry key change that needs to be made, or the specific mounting process that needs adjusting, etc. I will then be able to share it out to everyone who reports the issue.
Some sample things that a technical person might use to track down this issue are:
- Review system and Office logs for details that may give a clue as to why the system is not performing as desired
- Enable enhanced logging on the system while the problem is seen
- Compare logs from an affected system to a non affected system of similar configuartion
- Packet capture of a slow open or save
- Patch Office
- Re-Install Office
- Upgrade Office
- Patch the OS
- Upgrade the OS
- Rebuild the system in place
- Test a virtual machine running on an impacted system to see if it behaves the same or differently
- Test multiple accounts on the same system
- Note any differences based on time or network of a given system
- Move from a wired to a wireless connection (or vice versa)
- ?
In the meantime, the current workarounds for this issue are:
- Making a local copy of any Word documents, edit in place, and move back to the U Drive
- Use of Office 365 for Word documents
Recent events related to the U Drive:
- Samba upgrade for CVE-2017-7494
- The outage of 6/7
- Revert Samba upgrade for CVE-2017-7494
- The placement of rate limits on U Drive connections to protect the service from errant clients. A problem identified while debugging the outage.
- Installed mitigation against Samba security flaw CVE-2017-7494 since upgrade was rolled back.
- Increase to maximum winbind clients allowed (improved throughput)