

One security engineerโs trials and tribulations attempting to comprehend one of the least known but most powerful Windows services.
Before reading this post, please be sure to read @jepayneMSFTโs excellent post on Windows Event Forwarding: Monitoring what mattersโโโWindows Event Forwarding for everyone
Additionally, also check out Microsoftโs Use Windows Event Forwarding to help with intrusion detection
If youโre new to the concept of Windows Event Forwarding (WEF), the long story short is that a service exists in Windows where you can specify one or more servers to operate as Windows Event Log collectors. These collectors server as subscription managers and allow you to cherry pick which event logs you would like to collect from endpoints and the forwarded logs are then stored in buckets on the collectors. When using the Windows Event Forwarding service, the event logs are transferred natively over WinRM, which means you donโt have to worry about installing any sort of log forwarder software (Splunk/WinLogBeat/etc) on all of your endpoints to send logs to a centralized location.
Aside from the obvious benefit of not having to deploy any additional software, there are some great benefits to using WEF:
โข Events are encrypted via Kerberos by default
โข Subscriptions can be created as XML files and backed by versioning software like git (just be sure to validate the XML!)
โข New machines are automatically enrolled into the logging infrastructure after joining the domain
โข WEF can be configured in either push or pull mode
โข The forwarding interval can be modified
Although thereโs no shortage of documentation highlighting the benefits of using WEF, the primary focus of this writeup is to explain all of the components that need to exist for WEF to operate correctly and to document strategies for troubleshooting when things arenโt working as expected.
There are countless guides on how to configure WEF available online of varying quality. The core components of the installation are:
4. One or more event log subscriptions. A subscription is a collection of events based on Event IDโs or other criteria that tell the endpoints which event logs to forward. NSAโs Information Assurance group has an excellent set of subscriptions to use as a baseline:
https://github.com/iadgov/Event-Forwarding-Guidance/tree/master/Subscriptions/samples
5. A GPO to add the NETWORK SERVICE account to the Event Log Readers group.
6. A GPO to set ACLs on all relevant event log channels to allow read access by the Event Log Readers group. Many channels include that ACL by default, but the Security and other custom logs under the Microsoft/Windows service logs do not. More on creating that GPO here: https://support.microsoft.com/en-us/help/323076/how-to-set-event-log-security-locally-or-by-using-group-policy
7. Any organization who takes security seriously should have a GPO to enable enhanced auditing on windows enabled across their fleet: https://technet.microsoft.com/en-us/library/dn319056(v=ws.11).aspx
These seven items may sound relatively straightforward, but there are a surprising number of areas where itโs easy to make a mistake. Additionally, many aspects of WEF are unintuitive and make working with it less than user-friendly. In the next section, weโll cover the tools that are available to gain better insights into your WEF configuration.
Once youโve completed setup on your WEF infrastructure, you may find that hosts are not checking in, or certain events are not being forwarded. It can be confusing to understand why things are happening, but this section aims to give you the tools you need to effectively troubleshoot an installation.
On your WEF subscription manager you can see realtime statistics about your subscriptions via the Event Viewer console by selecting a subscription and clicking โRuntime statusโ:
Subscriptions can also be edited by choosing โProperties > Select Eventsโ. NOTE: Itโs possible to accidentally load subscriptions with broken XML via the wecutil command (ask me how I know).
wecutil (Windows Event Collector Utility) and wevtutil (Windows Event Utility) can also help you gain valuable insight into the current state of your subscriptions.
You can see the last time a host checked in to a specific subscription by running wecutil from a subscription manager:
PS C:\Windows> wecutil gr DNS
Subscription: DNS
RunTimeStatus: Active
LastError: 0
EventSources:
dc.windomain.local
RunTimeStatus: Active
LastError: 0
LastHeartbeatTime: 2017โ07โ22T07:54:27.296
TIP: To force a check in from a host, run gpupdate /force.
wevtutil is equally useful from the endpoint side to view things like event log channel ACLs:
PS C:\Windows> wevtutil get-log security
name: security
enabled: true
type: Admin
owningPublisher:
isolation: Custom
channelAccess: O:BAG:SYD:(A;;0xf0005;;;SY)(A;;0x5;;;BA)(A;;0x1;;;S-1โ5โ32โ573)(A;;0x1;;;S-1โ5โ20)
logging:
logFileName: %SystemRoot%\System32\Winevt\Logs\security.evtx
retention: false
autoBackup: false
maxSize: 20971520
publishing:
fileMax: 1
NOTE: If a channel you are attempting to pull events from does not contain the SID for Event Log Viewers (A;;0x1;;;S-1โ5โ32โ573), it will not be able to be read and an error message will be generated in the log channel documented below.
This channel is a great place to look when trying to determine why events from specific subscriptions arenโt coming in.
This log exists deep within the event viewer on each enrolled endpoint that logs windows event forwarding runtime status information. Unfortunately for us, the error codes displayed in these messages donโt seem to be documented anywhere. Fortunately for you, Iโve anecdotally determined what some of them mean.
Here are the following errors Iโve come across in this logfile:
Error Message: The subscription $name can not be created. The error code is 15008.
Issue: The subscription XML is malformed or the subscription is somehow invalid
Solution: Update the XML to be valid.
Error Message: The subscription $name is created, but one or more channels in the query could not be read at this time.
Issue: Event Log Reader group does not have read access to a channel specified in that subscription
Solution: Use wevtutil to view the current ACL on the relevant channels and create/update a GPO to allow read access to that log channel
Error Message: The subscription $name can not be created. The error code is 5004.
Issue: A channel specified in the description does not exist on the host
Solution: This is not necessarily a problem. If you have a subscription meant for DNS servers and only your Domain Controllers run DNS, you might see this error message on any server that doesnโt have the corresponding DNS server log channel.
Properly administering WEF infrastructure requires a wide range of skills. The user should have a solid understanding of Group Policy, Event Log ACLs, Advanced Auditing, Powershell, and Active Directory. Itโs tricky to properly configure WEF initially, but in my experience it has been rock solid and extremely performant. I highly recommend configuring WEF if you have a Windows domain in your environment and I hope this guide helps you work through any problems you may encounter along the way!
Create your free account to unlock your custom reading experience.