Security Scene: Know What's Targeting Your Network Infrastructure
Improving the accuracy of security monitoring is an evolving process that takes into account both updates to the operational environment being monitored, as well as hunting for emerging threats. Our security program for the national R&E network has improved the security monitoring capabilities of the network, progressing from basic break/fix monitoring for operational purposes, to adding the identification of potential security events which may represent malicious traffic or actions targeting our network infrastructure. While there are multiple sources of data often used for security monitoring, this article is about a sometimes- overlooked source of data, syslog from core packet-forwarding routers. This article discusses some of the security insights gleaned from syslog, hopefully encouraging more network service providers to incorporate security monitoring using syslog from their infrastructure as a security operations practice.
First, a simple description of our operating environment for security monitoring. Our core routing platform consists primarily of Juniper MX960 routers all running the same version of JUNOS. There are two feeds of syslog data from the routers, one is to the Internet2 NOC for break/fix monitoring and the second feed is encrypted and sent via syslog reflectors across Merit's network to the Internet2 Splunk indexer located in Ann Arbor, which decrypts the data and stores the events in Splunk. The security team uses a Splunk search head for routine reports and ad-hoc queries of the syslog data. Following are some examples of what was uncovered through analysis of syslog from our routers.
Developing a baseline of "normal" events occurring within the network can be helpful in identifying abnormal events when they occur. One of the events our routers syslog is a firewall filter 'discard'1 which happens when the router is the destination of packet that does not have a corresponding firewall filter rule accepting the packet. Here's a recent trendline showing the rate of discards per 10-minute periods:
Most of the packets discarded are involved in scanning IP address ranges which happen to include the IP addresses of the interfaces of our routers, falling within the range of about 60,000 discards per 10-minute period or 6,000 scans per minute. Knowing the baseline rate of discards allows us to look into increased activity and determine if our network infrastructure is at risk or not. Having a baseline of scanning activity also yields insight into emerging threats targeting networks in general. Similarly, a significant decrease in the discards per minute usually means that our something is broken with our monitoring.
In the fall of 2016, we began to see an increase in scanning of tcp/23 and tcp/2323. At the time we were unaware of cause for this increase but continued tracking it. Soon the source code for Mirai botnet was released2 and the reason for the scanning became immediately obvious, the scanning was related to searching for infected IoT devices. Here's a graph showing the frequency of scanning over the fall of 2016:
Since then, we have been able to notify campuses of potential Mirai botnet infections when we observe scanning of our network infrastructure originating from their networks.
Using a Splunk add-on called GeoASN, we are able to correlate source IP addresses with Autonomous System Numbers plus geographic location based on the MaxMind database. These correlations are sometimes imperfect and may require manual cross checking using whois and traceroute for confirmation. Below is routine report based on scanning of our routers showing source ASN, source country, destination port of our router, a sparkline indicating frequency over time, and a total count of discarded packets for the period. The AS numbers showing a red line after the name appeared in an Akamai State of Internet Security Report3 as top sources for DDoS reflection attacks. While the discards from these top sources of attacks were not service impacting to our operations, it does demonstrate the need for continued vigilance of security monitoring. Also in the report below, "AS87 Indiana University" appears high in the discard list. This is due to repeated and continuous reachability scanning of our network infrastructure performed by the Internet2 NOC as part of production monitoring.
Another example of syslog information providing insights into security threats occurred earlier this year when we noticed a significant increase in scanning activity originating from Russia. Typically, Russian sources of traffic recorded as discards is noticeable, but usually China is the top country for discarded traffic. But Russia was the largest source of scanning and we were unsure as to the motivations. Here is a graph showing discards by country that depicts the increased Russian activity (purple line) when compared with the top five countries of origin:
The increased scanning of our network infrastructure shortly preceded the US CERT Alert TA18-106A "Russian State-Sponsored Cyber Actors Targeting Network Infrastructure Devices"4 issued. As a result, we notified the Research and Education Network provider community of the advisory with evidence of reconnaissance of our infrastructure.
Future Security Scene articles will look into additional security monitoring topics and how they are used to identify threats within the Internet2 network. If you are doing some interesting work with security monitoring, I would like to hear from you. Please contact me and tell me what you're doing. We can share lessons learned with the community in a future blog post.