As we’ve built Scanner, we’ve learned a tremendous amount from our users about the important role that security logs play in detecting threats. Among the most insidious adversaries they face is the Advanced Persistent Threat (APT). These are complex, multi-staged, and stealthy cyber-attacks, sometimes sponsored by nation-states or cybercrime syndicates. Their primary aim is to gain unauthorized access to a network and maintain it undetected for an extended period, often months or even years. As a result, they can cause significant financial and reputational damage to companies.
According to a study by IBM and the Ponemon Institute, it takes an average of 277 days to identify and contain a breach. Furthermore, the global average cost of a data breach has risen to $4.35 million. This slow response and high cost underline the importance of adequate tools and strategies to combat APTs.
If your SIEM or log management tool can only realistically retain a few weeks of logs due to its high cost, then APTs can be challenging to detect and investigate, requiring manual, laborious digging through cold storage logs. Scanner avoids these log retention limitations by running on top of your S3 buckets and using serverless functions to traverse S3 index files at high speed, up to 1 TB per second. In this blog post, we will examine the crucial role of security logs and security teams in tackling these insidious threats, and we will show one example of how Scanner can detect APTs using queries that can quickly scan years of logs stored in S3.
Understanding the role of security logs
Security logs are a rich source of data generated by network devices, servers, databases, and other IT systems. They provide a chronological record of events, capturing who did what, where, and when. In essence, they serve as the ‘black box’ of your IT environment.
Accordingly, the significance of security logs in the battle against APTs cannot be overstated. Unlike traditional cyber threats, APTs are stealthy and slow. They typically avoid sudden or large-scale changes that could trigger alarms. Instead, they favor subtle, low-profile activities that blend into the normal ‘background noise’ of a network.
Thus, while an APT attack might not set off immediate alerts, it will still leave traces in your security logs. For instance, logs might show unusual patterns of network traffic, strange login times, or abnormal user behavior. By continually monitoring and analyzing these logs, you can spot these telltale signs of an APT attack and respond accordingly.
The key here, however, is that this monitoring needs to be both proactive and retrospective. Given the prolonged and stealthy nature of APTs, the signs of an attack might only become apparent over time. By looking back through your historical logs, you can connect the dots and uncover the true extent of the threat.
Detecting APTs with Scanner
Scanning CloudTrail logs
If you are running an application in AWS, it is useful to enable CloudTrail logs to record all API calls made in your environment. Since CloudTrail logs can be configured to be stored automatically in S3, you can use Scanner to analyze these logs in-place to look for suspicious activity.
For example, you can configure a detection rule to look for CloudTrail events indicating that multi-factor authentication has been disabled.
%ingest.source_type: "aws:cloudtrail"
and (eventName: "DeactivateMFADevice" or eventName: "DeleteVirtualMFADevice")
| stats count by sourceIPAddress userIdentity.userName userIdentity.accessKeyId
Disabling multi-factor auth could be a legitimate action, but it could also indicate that an AWS IAM user account has been compromised. Usefully, the query above will reveal the source IP address, user name, and AWS access key of the actor. This gives you some identifying details that you can use to investigate further and verify whether there is a real security threat.
For example, let’s say that the detection rule above was triggered. As a result, you received a Slack notification that multi-factor auth was disabled. The notification contains these fields: sourceIPAddress: "153.60.4.182", userIdentity.userName: "jane-smith", and userIdentity.accessKeyId: "AKIAIOSFODNN7EXAMPLE". With these values in hand, you can use saved searches in Scanner to uncover more suspicious activity.
Detecting suspicious AWS API calls
For example, you might leverage a saved search to look for instances where the suspicious user made any of the following API calls:
- Create new AWS access keys. By creating new keys, the attacker can maintain access to your system even if you shut off the original compromised account.
- Delete CloudTrail logs. The attacker can delete logs to cover their tracks.
- Delete an IAM policy. By deleting IAM policies, the attacker can gain access to more systems because the policies can contain important permissions checks.
- Create an IAM policy with full admin access. Creating a new IAM policy with full wildcard access
"*"can give the attacker access to more systems.
%ingest.source_type: "aws:cloudtrail"
and (sourceIPAddress: "153.60.4.182"
or userIdentity.userName: "jane-smith"
or userIdentity.accessKeyId: "AKIAIOSFODNN7EXAMPLE")
and (eventName: "CreateAccessKey"
or eventName: "DeleteTrail"
or eventName: "DeletePolicy"
or (eventName: "CreatePolicyVersion"
and requestParameters.policyDocument.statements[0].action: "*"))
You can also run queries in Scanner to detect data exfiltration. For instance, you can look for all API calls related to S3 or DynamoDB from the suspicious user. A large number of S3 GetObject calls or DynamoDB Query or Scan calls could indicate data exfiltration:
%ingest.source_type: "aws:cloudtrail"
and (sourceIPAddress: "153.60.4.182"
or userIdentity.userName: "jane-smith"
or userIdentity.accessKeyId: "AKIAIOSFODNN7EXAMPLE")
and (eventSource: "s3.*" or eventSource: "dynamodb.*")
| stats count by eventSource eventName
Scanning Github Audit logs
The most costly form of a data breach comes from compromised credentials, which takes on average 327 days to identify. In fact, one of the worst breaches that my co-founder, Steven Wu, and I have seen first-hand was a credential breach at Cisco shortly after our team was acquired there. A disgruntled, former Cisco employee used their AWS credentials to destroy over 400 virtual machines and wipe out 16,000 Webex accounts. Hence, it is essential to maintain tight control over credentials and immediately take action when credential misuse is detected.
One common way that credentials are leaked is when developers commit secrets to Github repositories. With this in mind, if you are shipping Github audit logs to S3 and are analyzing them with Scanner, you can create a detection rule for credential breaches by looking for the secret_scanning_alert.create event generated by Github:
%ingest.source_type: "github:audit"
and action: "secret_scanning_alert.create"
Additionally, you can detect when an attacker is trying to cover their tracks by looking for events where Github security features are disabled, bypassed or deleted:
%ingest.source_type: "github:audit"
and (action: "*secret_scanning*.disable"
or action: "*secret_scanning*.bypass"
or action: "*secret_scanning*.delete"
or action: "*vulnerability*.delete"
or action: "dependabot*.disable")
If these queries yield results, you will see Github audit log events that will give you more information on the suspicious actor, which repositories they are accessing, and when their actions occurred. Consequentially, this can help you determine the impact and blast radius of a potential breach, giving you the information you need to take action by revoking user access, disabling leaked credentials, and more.
Look for the suspicious user across all of your logs
You can use Scanner to search for the suspicious user’s activity across all of your log sources without needing to know what each log source’s schema looks like. To illustrate, you can search for all activity containing the suspicious IP address, user name, or AWS access key across all of your log sources with this simple query:
"153.60.4.182" or "jane-smith" or "AKIAIOSFODNN7EXAMPLE"
Since Scanner can quickly analyze years of logs (~1 petabyte) by leveraging the index files it stores in your S3 bucket, it can help your team discover APTs that engage in malicious activity over time. For example, if an APT executes one suspicious API call per month, traditional SIEMs that only retain one month of logs may miss the long-term pattern and allow the APT to slip by unnoticed. We designed Scanner to avoid these unnecessary log retention limitations and reach much higher scale using S3 for storage and serverless functions for fast index file traversal during searches.
The critical role of security teams
While security logs provide the raw data, it is your security team who transforms this data into actionable intelligence. These individuals are at the forefront of your defense against APTs. They are responsible for monitoring your logs, detecting threats, responding to incidents, and ensuring the ongoing security of your systems.
Given the complex and evolving nature of APTs, security teams need to be skilled, agile, and continuously learning. For example, they need to understand the current threat landscape and the tactics, techniques, and procedures (TTPs) used by APT groups. They should also be adept at using various security tools.
To illustrate, here are four key roles and responsibilities a security team might have in tackling APTs.
#1 – Proactive threat hunting
Rather than waiting for alerts, security teams need to actively search for threats. This involves continuously monitoring logs, analyzing anomalies, and investigating suspicious activities. By doing so, they can uncover hidden threats and neutralize them before they cause significant damage. SIEM tools like Splunk Enterprise Security and QRadar can be useful here, but their costs are quite high if you want to retain months or years of logs. For this reason, our users have found Scanner to be a strong complement to Splunk or QRadar. Specifically, a common scenario we see is the retention of one week of logs in Splunk and a full year of logs in S3 where they can be analyzed and queried by Scanner.
#2 – Incident response
If an APT is detected, the security team is responsible for responding quickly and effectively. This includes isolating affected systems, eradicating the threat, and restoring normal operations. They also need to analyze the attack and learn lessons to prevent similar incidents in the future. Consider using response automation tools like Tines to respond rapidly to incidents, and propagate alerts to your team quickly with Slack integrations and PagerDuty.
#3 – Threat intelligence
Knowledge is power in the world of cybersecurity. The more you understand about APT groups and their tactics, the better equipped you are to defend against them. Accordingly, security teams should regularly gather, analyze, and share threat intelligence. This can help them anticipate threats, refine their defenses, and respond more effectively to incidents. Staying on top of the CISA advisories feed is useful, and we recommend checking out resources like Herman Slatman’s popular awesome-threat-intelligence repository on Github.
#4 – Employee training
APTs often use social engineering tactics, such as phishing, to gain initial access to a network. Therefore, security teams have a vital role in educating employees about these tactics and promoting safe online behavior. Regular training sessions can significantly reduce the risk of successful APT infiltration. For example, you might try out courses from KnowBe4, which is one of the fastest growing security training companies out there.
Building a resilient defense
In the face of APTs, organizations must adopt a multi-layered, proactive security strategy. Security logs and security teams are crucial elements of this strategy.
Security logs provide a wealth of data about your IT environment, serving as an invaluable resource for threat detection and investigation. Meanwhile, your security team transforms this data into actionable intelligence, using their skills and expertise to detect and neutralize threats.
However, combatting APTs also requires advanced security tools, continuous learning, and a culture of security awareness. It requires a commitment to ongoing investment, not only in technologies but also in people. While the costs may be substantial, the potential costs of an APT attack – both financial and reputational – make it a worthwhile investment.
As the cybersecurity landscape continues to evolve, so too must our strategies and defenses. By leveraging the power of security logs and strengthening the capabilities of security teams, organizations can stand strong against APTs and ensure the ongoing security of their systems and data.
Share this article
Scanner is a security data lake tool designed for rapid search and detection over petabyte-scale log data sets in AWS S3. With highly efficient serverless compute, it is 10x cheaper than tools like Splunk and Datadog for large log volumes and 100x faster than tools like AWS Athena.
Cliff Crosland
CEO, Co-founder
Scanner, Inc.
Cliff is the CEO and co-founder of Scanner.dev, a security data lake product built for scale, speed, and cost efficiency. Prior to founding Scanner, he was a Principal Engineer at Cisco where he led the backend infrastructure team for the Webex People Graph. He was also the engineering lead for the data platform team at Accompany before its acquisition by Cisco. He has a love-hate relationship with Rust, but it’s mostly love these days.