Category: Security Staffing



This year I was interviewed by Infotech Research Group pertaining to the difference between a SOC and SOF for a solution they were developing collateral.  Prior to the interview I prepared a response to their questions.  The data they were asking is very similar to questions I’m asked regularly around security operations solutions.  Therefore, I wanted to take this opportunity to put out some content around the core topics of security operations centers and functions. 

The fundamentals

What is the different between a Security Operations Center (SOC) and a Security Operations Function (SOF) and what is required to provide a state of the art intelligent security operations function?

To begin we must first clarify the difference between the two concepts.  In general a SOC is or can be a portion of the overall SOF.  A traditional SOC will usually focus around a SIEM whereas today’s security operations need to integrate multiple products and automate intelligence and response in real time. The traditional SOC was previously a large physical security command center.  Today the SOC has shifted into more of a function where many responsibilities are spread outside of the traditional physical command center.

We should not forget that one advantage of housing the security operation within the same physical location is the capability to easily segment, secure, and control the data within the environment.  When you start opening up the function you also open up access making it harder to secure the security data in larger environments.

1.0 and 2.0 what is the difference?

What are the key differences between a traditional SOC or SOF and the next generation solution (or version 2.0) and how is the next generation going to handle the newer technologies, as well as the continual changing threat landscape associated with big data, mobile computing, and social media?

The big difference is that monitoring is now no longer contained to inside organization and in many cases the data is not either, so in many cases you will be relying on third party organizations more.  That means you must have good processes in place with defined SLA’s to understand the capabilities of each provider.  Having a good understanding of the security for that third party environment will be critical to ensuring your data is not leaked.  Also having a good way to detect malware in the environment will also be important to help prevent lateral movement across the organization and its suppliers or providers.

Also, because of the advanced threats that specifically target individual personnel the next generation of security operations solution will eventually have to enter more into the home environment to protect those key targets.  Key executives that are targeted through their children or other family members will need to have some protections in place that help mitigate the risk.  These advanced threat actors will compromise another family member’s account by sending an email with a title like “homework help” or something similar to fool the victim.  Little does the parent know this was an attacker who hacked their kids account for the entire purpose of sending a word document with malware to attack the target executives work account eventually.

Some other items that will come more into play is real time threat intelligence.  Getting automated feeds in formats like STIX or open IOC with specific data that is relevant to your organization will become the norm for large enterprises.  In some cases even a private intelligence cloud will be used.

Why Implement a SOC or SOF in the first place?

Many of you may not currently even have a security operations function in place, however due to recent attacks you are wondering if this is the right fit for your organization.  To help with that decision understand there are usually two main drivers.

  • One is that the organization has had a series of major incidents and as a result needs to implement some type of monitoring and protection function.
  • The second driver is usually for increased profits.  The large MSSP or Telecommunications organizations will implement models to sell the managed SOF to their clients and therefore will build out operations.

At some point you really need to understand the threats to your organization.  Without a clear indication of the attacks the organization faces daily it will be hard to justify the cost of a SOC/SOF and therefore you may need to consider an outsourced option.

Responsibilities of the SOF

We know that security monitoring and incident response are obviously included at some level, but how about security devices maintenance, identity and access provisioning/de-provisioning, even security solution design and implementation.  Should those all be included?

Personally I don’t think device maintenance is a good function of the SOF.  My experience in the field having done penetration tests against organizations that have MSSP’s (Managed Security Service Provider) who perform both the security and maintain functions show that many items are typically neglected.  In short, the MSSPs can do everything from monitoring to maintenance, but in many cases they end up doing a poor job at everything because there is a lack of focus on one task.  For device management and maintenance the SOF should really only be involved with reviewing security compliance around device maintenance and involved in coaching or developing the processes to ensure out of compliance items are either removed from the network until updated.

Other functions such as identity access and entitlement reviews can be a part of the SOF, but usually this is not included based on a decision around corporate cultural.  Access control in many cases resides outside of the SOF or in an isolated SOF department and unless the organization can manage the cultural issues it will not work effectively in the SOF.  The SOFs function really should be to review and compliance check on access controls and violations of access.  We also need to be careful about giving the SOC too much capability because then the SOC becomes a single point of failure for security.

Overall some core items that should be provided by the SOF are:

  • Monitoring, alerting, threat analysis, correlation and intelligence
  • Incident response, investigation, malware and threat analysis, and serving as an extension of the forensic teams in some cases
  • Advisory on corporate security solution designs

Outsourcing the SOF

How do you determine then if the SOF should be outsourced or not?  There are several drivers in the decision making process that will determine if the function should be outsourced or not.  In almost all cases this is relative to cost or that fact that the organization does not have the appropriate skills in house.  In some cases the organization may also decide that it should not be investing in services that deviate from the core business goals.

Leveraging Centralized or Distributed Response Models

A challenge in many security functions is to determine the correct response model.  This really depends on the global extent and cultural diversity of the organization.  If the organization is global there will be many challenges if a central response team is implemented.  If you have to send someone onsite half way around the world it could take days depending on where in the world that person must fly.  Also you will run into challenges around language and VISA requirements.

On the other hand having a team that is distributed and speaks or reads multiple different languages is will enhance any response team in providing a timely and adequate response.  The time to location is quicker, command and control can be done while onsite (war room), and intelligence can be shared via the coordinating or central lead team.

Key Aspects of the SOF – People, Process, and Technology

To implement and maintain a successful SOF the right defense in depth strategy is required.

People – A successful SOF must have skilled staff that can think like the adversary.  This staff must also have technical knowledge and troubleshooting abilities to understand the threats and attacks.

Technology – The technology strategy at the most basic level must have several core components.  New generation malware detection at the network egress points and endpoints is a requirement.  A SIEM or correlation engine is necessary to integrate the logs of many technologies.  Finally some custom integration between the core network, endpoint, and SIEM products is required to automate as much as possible for both identification and containment of the threat.  Threat intelligence feeds will also become a core item in the next generation SOF to provide awareness of the latest trends as well as threats to the industry and its service providers.  Most current SOC functions try to increase the function with more analysis software.  The mistake with that is it requires the time of your skilled staff where as they should be looking to automate the analysis and containment tasks as much as possible and use a series of base products required to stop the attacks.  Some other SOF based solutions will talk about risk-based decision systems.  This is really just correlation and automation of high risk threats.

Process – For process there are several components.  Sound roles and responsibilities must be defined.  Automation of trouble ticking and remediation processes need to be in place. Threat risk assessments must be conducted to identify critical targets and the threat events that are high risk to the organization.  A SOF at many levels must be integrated into every important aspect of the business.  For example, if a hurricane is coming and the BCP department says we are on hurricane watch.  The SOF should be looking for phishing attempts leveraging the hurricane as an event to try and attack the organization suspecting that employees will be reading every email and website about the hurricane. 

Challenges within a SOF

There are hundreds of challenges to run a top notch next generation SOF, but there are some fundamentals that must be addressed.  Education of staff is critical.  Without the right skills the attackers will always hard to find in the organization and even harder to remove.  Ensuring that the operators follow the procedures with little or no exception and continually update information and tactics based on the threats facing the organization is important.  The management of the SOF must also make sure executive management understands the extent of the threat otherwise funding and other critical controls will be neglected or overlooked.  Also, everyone talks about implementing more process in the next generation SOF, but in reality the real solution is to take as much human interaction out of the process and automate analysis and protection. A solution for querying the endpoint and enforcing rules based on network detections, a solution for leveraging and automating multiple technologies, and a solution for using first detection malware appliances to forward blocking information into Firewalls and Web/email gateways; these are the critical improvements and challenges required to build out the next generation SOF.

The 5 People, Process, and Technology Requirements

There are various sizes of organizations and every SOF will be different in many ways, but with respect to people, process, and technology the simple and most effective measures for providing security to an organization are:

Technology

  1. Egress malware blocking technology covering both email and web vectors combined with a web proxy and spam filter.
  2. Application blocking and anti-virus software on the endpoints
  3. SIEM for centralized logging and correlation of information
  4. Global Risk and Compliance software for integrating security with other processes within the organization
  5. After that you will augment these core components with other software for compliance and other business requirements.

People

  1. Strong leadership
  2. A strong person in network and application security
  3. A strong person in risk management and security policy
  4. A strong malware and forensic skill on staff
  5. After that you can build out the team based on the scope and mission of the security function and leverage contractors or outsourced solutions.

Process

  1. Automation of as much process as possible
  2. A strong set of core policies and procedures for change control, incident response, alerting and reporting that focus on protecting the organization and its mission.
  3. Collection of metrics
  4. A process that modifies regularly to reduce detection, containment and remediation time
  5. A process to understand the real threats to the organization

Measuring Effectiveness

As a person running a SOF you will always be asked to prove the effectiveness.  Is there some important Key Performance Indicators (KPI) to help prove the value of the SOF?  This is a difficult question and each organization may have specific KPIs based on the goals of the organization.  However in general there are some core items that should be measured.

People

To help make sure each member of the SOF is working effectively, metrics around the roles and responsibilities for individual is important to measure.  This is important to help measure the skill level of each person and that each individual is working toward the mission of the SOF.  Therefore, measuring items such as:

  • Shift logs and components captured in shift logs
  • Hours analyzing events, hours automating, and hours researching

Together these items will help determine what you need to focus spending on and to help free up resource time in the future.

Technology

There are several items around the technology to measure including:

  • How many incidents over different time metrics (week/month, etc.)
  • Amount of incidents and events detected and the percentage of those that were automatically blocked or contained
  • Timeline breakdown for each incident (When it was detected, contained, remediated)
  • The breakdown of technology detection; there will always be overlap in detection, so this overlap should be measured to help effectively determine if a control is needed

Process

In the process area you will want to track several many different things to prove the effectiveness around the incident response process including:

  • Amount of time to resolve an incident
  • Estimated cost to resolve an incident
  • Increase or decrease in security spending over time (compare against protection metrics)

One of the other items that gets overlooked sometimes it to track the time it takes collect and report on the metrics.  Inevitably, a large portion of the documentation is manual, and collecting metrics is manual due executive presentable formats not existing.  The SOF will spend days at the end of each reporting period to generate the metrics and report to management.  If you can automate the collection and reporting by using a global risk compliance system, some sort of ticketing system, or by using some custom code then that will help immensely in the long run.


Security technology spending is at an all-time high.  Determining the right strategy to reduce cost is essential to security planning and any CISO’s agenda.  Consider implementing the 7 following ways to optimize new security technology investments.

  1. Consolidating security vendors for particular solutions will reduce cost as a result of volume discounts and costs associated with increased complexity, risk, down time and staff management.
  2. Reducing the number of endpoint agents will reduce cost by decreasing the complexity of the environment, testing cycles per agents, and administrative staff time required.
  3. Negotiating hardware and software licensing costs as well as security professional services for longer periods of time allows security vendors to reduce paperwork and management costs, which in turn can be provided back to the organization.
  4. Implementing adequate security protections can reduce costs associated with employee productivity loss, security breaches, as well as the IT labor costs associated with endpoint infections, managing signatures, false positives, tuning, etc.
  5. Using automated software to provide agent updates, tuning, patches, and signatures reduces costs associated with employee productivity loss and IT labor management.
  6. Reducing complexity of the environment by consolidating consoles for items such as endpoint and network technology, logging, or security configuration management provides faster access to relative data and possible security incidents.  Less complexity and faster access reduce costs by decreasing the infection rate and reducing the IT labor management.
  7. Focusing on the primary business while outsourcing certain security functions should be evaluated regularly.  Some costs may be reduced by avoiding security infrastructure and software costs as well as additional IT labor and training costs.

Whether defending against common malware or some determined Nation State, being able to proactively detect attacks and changes in the organization are required.  The past year I spent a large amount of time helping several organizations setup and put in place the right people, processes, and technology to help defend against increasing security threats.  Although many organizations spend millions of dollars on technology and hire staff to monitor security 24/7 the organizations were still lacking two fundamental items.

  1. The people although good at monitoring lacked the attack and threat mind set.  The staff was not able to figure out when an actual attack was happening.
  2. Second the organizations lacked the basic security operations processes required to keep track and make appropriate use of the vast amounts of data.

As a result I spent the past few months developing a whitepaper that specifically addresses the primary components of a SOC, which can be used to help organizations setup a centralized core and embark on developing the correct operational processes.  Although I don’t address item number one above, this paper explains in detail the following.

  • Defining the SOC
  • Determining the Processes
  • Understanding the Environment that needs protected
  • Identifying the SOC Customers
  • Staffing the SOC
  • Managing the Events
  • Leveraging ITIL compliance

Creating and Maintaining a SOC – The details behind successful Security Operations Centers

If your organization is under attack and you have invested in more people and technology be sure to implement the right processes and build a foundation for future defense.


There has been a large amount of security information and recent attacks posted in the media.  We have Mandiant’s report on China as well as several issues concerning Java.  The pure volume of information over the past year has made it difficult to keep up without a combination sources.  As a result InfoSecAlways has done a few modifications to the site.  Please check out the new “Security Feeds” in the right column (4th Block Down).  This is a combination of about 20 different security RSS feeds piping into the blog now.  You can check the site daily to get the latest news and updates in the industry.

Also, check out the links page as there are several new Threat and Vulnerability links added.  These are great if you are looking for specific attacks, breaches, or threats.


I have been using this blog to track a good amount of search hits looking for security staffing and governance.  Unfortunately when you search there is not much out on the Internet.  If anyone is interested let me know and I will start an open source project off this blog to create a governance and staffing solution/program.

For those that have little or no knowledge in this area I suggest you review the Security Task Force documentation and the EDUCAUSE updates located here:

EDUCAUSE Information Security Governance Assessment Tool

For an open source program I would like to build of the current work, but also provide a lot more emphasis on the organizational charts and the roles and responsibilities.  If you are interested please let me know and we can get everyone together and create an updated model for multiple industries.


The polls are open!

While visiting this site please check out the new IS Management page and contribute to the voting polls.

If you would like to see new or different polls added let me know.


This whitepaper has a good overview of key components of a risk based security plan, which have been put into practice on several occasions.  This provides good direction with a decent amount of detail.

Site Requires Registration:

The document is about 12 pages explaining the steps for performing a risk assessment to developing a security plan to determining security budget.


One of the key problems a security manger must tackle is defining the budget for security training.  Many awareness program guides break it out into a method similar to the following:

 

  1. Identify security roles and responsibilities
  2. Conduct a needs assessment
  3. Identify the gaps
  4. Develop and implement the training plan

 

Skills Identification

The key step here is the identification of roles and responsibilities.  Identification of security roles and responsibilities is probably one of the most important fundamental aspects to a successful security program.  Although, writing sample roles and responsibilities or breaking out each of the above steps is not the focus of this topic, it is important when defining the core security staff’s training to build on the role definitions by creating a skills identification table.  A skills identification table will work for most organizations because it provides a quick profile of each security professional.  To create a skills identification use excel or a similar program and setup a structure similar to the one shown in the table below.

 

 

List each employee in the security program in the left column and then ask each one of them to fill in their certifications and training.  Columns should be added for all security certifications and training associated with employees.  This information will provide the security leader with the organizations current security capabilities.  It will also be easier for the security leader to assign the appropriate personnel to security issues based on their training and certifications.  For career planning you could also expand this model to include a section for desired certifications, training, or expertise.

 

Applying to Budget 

Now that each employee has provided their information the identification table can be used to help with the annual training budget.  Ideally the security leader should set the annual training budget for at least one training session a year for each employee.  The security leader should also take one training a year, but if cost becomes an issue than offset the security leader training by attending conferences and conventions.  If possible training schedules and classes can be used to prepare for new corporate projects by attending training with specific project needs.  Otherwise training should be defined with each employee based on their career goals and the goals of the organization.

 

Depending on the size of the core security team an average week of training may cost anywhere from $2500 to $5000 depending on location and accommodations.  To define and annual budget take the number of staff and budget for the $5,000 per person annually.  For example, 5 core security staff should have an annual budget of $25,000 dedicated solely to security training.  Determining the actual classes beforehand will help predict the budget more accurately and possibly save costs on travel.  If you are in a large organization, especially one that is decentralized the budget may increase significantly.  One way to reduce the cost is to identify key security gaps, such as application security, and pay for onsite training.  In this situation budgeting will have to be performed by contacting a vendor(s) to obtain pricing quotes.  Keep in mind there may be an issue with taking a large amount of employees away from their regular work. 

 

Overall there are several advantages to this staffing and budgeting approach.  One immediate advantage of increasing the security training may be reduced consulting costs.  Another advantage will be increased employee morale, as well as improvement of overall security.


Its about time!  Foundstone Professional Services has been added to the Avert Labs research blog.  So now the makers of all the free hacking tools are accessible online.  Check it out there are already some great posts. 

 http://www.avertlabs.com/research/blog/index.php/category/foundstone/

I’ve also added it as a Blogroll.


There is an article on The Register web site claiming security spending has soared to 20% of the IT budget.  This is based on a poll of 1070 organizations.

http://www.theregister.co.uk/2007/10/11/comptia_security_survey/

It is a shame the article doesn’t provide more detail.  It would be nice to know the industries surveyed, size of the organizations, and all of the categories assessed.  Does this review include staffing, business continuity, disaster recovery, Application security, etc.?

My experience shows that most organizations can’t account for the actual security dollars spent.  When evaluating IT security within an organization, excluding physical security and business continuity, most organizations I review are in the 1% to 5% range of the IT budget with the exception of the major financial firms and a few others.  These numbers are also pretty much in line with the CSI/FBI annual surveys conducted.

  • What is your experience? 
  • Can you account for your total security budget? 
  • What does that budget include?

Unfortunately this area of security is still lacking in the amount of free information available to the public and many of the assessments are limited to less than 1000 respondents.  I would be happy to post some links on this site if anyone has some good free resources or whitepapers.

Follow

Get every new post delivered to your Inbox.