Network Incident -> Groups Incident -> UWWI incident

Some UWWI groups are not currently in complete sync with the Groups Service

What and When:

This past weekend there was a significant network incident. That led to a Groups Service incident. The impact of the Groups Service incident was that change notifications for groups changed during that part of the weekend were never sent to UWWI.

UWWI also had an incident because of the network incident and had to restart our group agent, but that only delayed the processing of group change notifications. However, most of the group change notifications were missing in action because the Groups Service was had its own incident.

9172 groups were changed during that period, but due to our efforts, there are now less than 7100 UWWI groups which are out of sync from the Groups Service. Affected course groups have already been fixed.

What You Need to Do:

Be aware that there may be some slight group inconsistencies in UWWI for a little while longer.

More Info:

We have a standard way of resyncing groups which are out of sync, and once a month every group is subjected to this examination to ensure that no group falls out of synchronicity. This process is more resource intensive (it examines the state, figures out what is missing, and then fixes it, instead of just applying the changes). We’re selectively applying this to the group changes that went missing during the Groups Service incident, but it’ll take a bit for that to reach completion.

There are ~91,000 total groups in UWWI so this affected about 1/10 of all UWWI groups.

Brian Arkills

UW Windows Infrastructure Service Manager