Introduction
This is the fourth post of our blog series covering the key findings of our first research report “Top Challenges in Cyber Investigations & Recommendations for SecOps Leaders”, published on September 10, 2024. You can read the first two blog posts of this series here:
In this post, we will cover the third and final key finding of this research:
Investigations lack consistency, documentation and auditability
The interview process revealed well-known issues along with new patterns for enterprise investigations at scale. A lack of standardization in the investigation process, access to data and collaboration surfaced as agreed facts: 92% of respondents cited the lack of a standardized collaboration tool as a key challenge during cyber investigations. Reliance on inadequate solutions leads to inefficiencies, miscommunications, and loss of data.
Project communication is particularly important in onboarding new team members to an investigation, handing-off investigations between analysts, collaborating with subject matter experts and asset owners. Getting everyone on the same page becomes even more challenging when coordinating with external personnel. Problems begin to compound when pressures mount internally from management or external parties for frequent updates on current cases. External pressure from press, law enforcement, or other regulators may continue to escalate the situation, resulting in errors during investigations.
80% of CISOs find tracking and complying with regulatory reporting overly complex. This was especially true for organizations operating in multiple jurisdictions. Not having a standard for the investigation process, output and outcomes are contributing factors to complexity. Challenges in getting traceability and auditability of past investigations is the intersection of GRC, security operations, identity management, etc. There’s a lot of progress to be made on all fronts to truly overcome this chokepoint.
Reports are the standard medium to communicate the investigation process, outcomes and recommendations. Yet, building technically accurate reports that speak to technical and business audiences is a rare skillset. Conducting thorough and accurate investigations requires a deeply technical skillset. Whereas writing thoughtful reports of the technical process and its business impact requires social and writing skills which are not always common among technologists. This is why reporting is a daunting task for analysts of all levels. The research respondents confirmed this: 79% of respondents cited time-consuming reporting requirements and updating management (as well as other stakeholders) as a significant challenge. As a result, reporting becomes a time suck for investigators, and report outputs may not meet expectations.
This comes as no surprise to me, as report writing has always been my least favorite part of an engagement.
The dynamic and curious nature of investigations also comes with its own challenges: 72% of respondents found investigation scope creep problematic. Keeping the investigation focused is a constant challenge as analysts must continually evaluate the relevance of new information. The scope of an investigation may expand rapidly as new data is discovered. This scope creep complicates building a clear and concise investigation narrative. To limit this effect, investigators must have well-defined practices when distinguishing between crucial data points and counterproductive rabbit holes. These well-defined practices should guide the analysts for consistency, but the tooling must still allow the flexibility to avoid rigidity and allow analysts to follow contextual evidence that may be present.
At the individual level, we get better at things by accumulating experience through practice. At the organizational level, the only way to build institutional knowledge is by leveraging lessons learned in past activities to improve the process for future tasks. While this is common knowledge, implementing this for security operations does not appear to be common today. 69% of survey respondents did not programmatically link learnings from prior investigations. Past incident data is invaluable as a reference for both future investigations and as useful case studies to train new investigators. A powerful practice for training new resources is the systematic review of prior incidents and associated investigations. These post-mortems not only help the analyst to learn the organization’s environment but also help to teach investigative techniques. Through effective training, analysts will better understand common attack methods and the associated organizational protocols. All these elements are helpful both for training new investigators and refreshing the hands-on knowledge of the existing team. The result is better overall decision-making during a real incident for both inexperienced and seasoned investigators.
Command Zero’s perspective
We live in a world where tier-2+ analysts need to be jacks of all trades to run complex investigations. They need:
- Administrator level technology expertise on all systems within scope.
- Direct administrator level access to all investigated systems.
- Advanced cyber investigations expertise.
- The current and historical context of the environment.
- Advanced written and verbal communication skills to communicate with teammates, other technical teams, business and legal teams.
Add in the complexity of enterprise IT environments and sophisticated attacks, and it’s clear: cyber investigations need more structure to be repeatable and scalable. Best investigators are a rare breed who combine sophisticated technical and communication skills with excellent knowledge about the organization and the IT environment.
To get to the bottom of a complex investigation, an investigator needs 6-12 different tools and 3-8 hours to reach a verdict on average (Source: ESG The State of the SOC). Experienced analysts are hard to find and harder to keep. Today’s threat volumes and mature processes for detection result in more escalated cases. And each escalated case requires thorough investigations. As a result, many escalations end up without a deeper look. And many investigations that get started end up unfinished (without definitive verdicts within an acceptable timeframe).
So, how do the best investigators do it? For seasoned analysts, investigations are a manual process that combines manual controls, script bundles compiled over time and ad hoc communication. This is a model that works for that individual, but it is not repeatable, auditable and it certainly does not scale. The core of the process and the ever-growing knowledge is limited to experienced individuals and no institutional knowledge is built on past investigations.
Let’s admit it, as an industry, we haven’t advanced how we do investigations from the era of super admins. This was an era where all IT systems were kept in a single data center with total control, and a handful of super admins could know and access all systems within scope. The reality is that the IT systems and enterprise environments we investigate have evolved dramatically since then. “Super admins” as a concept is a thing of the past in today’s hybrid world. All distributed systems have dedicated administrators who only cover parts of the IT ecosystem, likely without the access or expertise for adjacent systems.
To adapt to today’s distributed IT environments, we need to shift how we do cyber investigations. We need a solution that keeps analysts in the driver’s seat while reducing the manual toil of the process through automation. We can deliver the best investigation outcomes only if we can provide the subject matter expertise and access for all systems to all analysts. Democratizing these capabilities will boost the confidence of each investigator and build a path for standardized investigation processes.
SEC’s recent Cybersecurity Disclosure mandates communication of cybersecurity incidents in four business days after the incident is determined to be material. The same statement also brings annual disclosure responsibilities for cybersecurity risk management, strategy, and governance. Being able to determine when there is an incident and documenting the analysis are important capabilities for all organizations. With this new statement, standardized, effective investigation practices and predictable documentation become regulatory requirements for public companies.
We can build standard processes for cyber investigations only if we can empower all tier-2+ analysts to conduct expert investigations. These processes can include how to collaborate and communicate during investigations. This approach would also solve for reporting, communications and scope creep issues noted about investigations above.
Recommendations
- Building detailed processes for cyber investigations is critical to success in complex organizations. As these processes develop, we need to consider three pillars that make an investigation:
- Technical investigation flow: Cover the technical documentation, access protocols and the analyst qualifications. Add lessons learned from previous investigations and organizational context when applicable.
- Collaboration during an investigation: Cover how multiple analysts, multiple business units and external parties can collaborate during an investigation. Defining how often they will communicate, and preferred communication methods are important.
- Reporting and communicating outcomes: Cover best practices for reporting and sample reports showing what good looks like to guide analysts with the reporting process. A library of quality reports always comes in handy for analysts looking for inspiration. (We are all totally doing this, right?)
- Look for ways to communicate lessons learned from past investigations in a structured way. Communicating high level flows, the decisions that affected the outcome, and identified areas of improvement with the analyst team helps foster a culture of sharing. Documenting these learnings and optimizing the process and system configurations (including detection, alerting, correlating and investigation tools) will drive continuous improvement. A weekly or monthly investigations office hour can be the right forum for these learnings, accompanied by a weekly/monthly written update.
- Keeping a list of compliance requirements and tagging investigations with the relevant compliance frameworks from inception is a best practice that will help gather the right information, present and communicate it in the right way. This proactive approach will save many cycles for the team as they can now run investigations to satisfy compliance/regulatory requirements.
Conclusion & What’s Next
We covered the third and final key finding of this research on this blog post, we will dig into the overall conclusion and recommendations on our next blog post in this series.
If you’d like to read the full report, you can download a copy from the report overview page on our website.
