The Information Security Consolidated Event Management (ICE) system is a more than 30-terabyte data warehouse used by the Microsoft Information Security team to analyze network utilization events captured by various sources, including over 100 proxy servers, mail servers, Net logon servers, etc. The ICE database processes approximately 1 terabyte of log data each day and it has become a key component in the incident response process, in addition to forensics investigations. Analysis of the proxy data has empowered the Microsoft Information Security team to identify and remediate numerous security issues that would have gone undetected otherwise.
ICE version 4.0 is an ambitious project set to deliver almost real-time data and high-query performance to the security team using Microsoft SQL Server 2008. Moreover, ICE 4.0 is also designed to perform all sorts of data filtering and transformation during the data-loading process, so the schema of data stored in ICE is tailored for investigation analysis/reporting needs. An Online Analytical Processing (OLAP) cube is built on top of the ICE data warehouse to facilitate aggregated queries. Join this session to learn how enhancements to the Microsoft SQL Server Integration Service (SSIS) 2008 dataflow engine have significantly improved the performance of loading, filtering and transforming 1 terabyte of network log data into the ICE data warehouse.
You can download the interviews here: WMA | MP3 Hi | MP3 Low
About denny.lee
Denny Lee is a Senior Program Manager based out of Redmond, WA in the SQLCAT Best Practices Team. He has more than 12 years experience as a developer and consultant implementing software solutions to complex OLTP and data warehousing problems. His industry experience includes accounting, human resources, automotive, retail, web analytics, telecommunications, and healthcare. He had helped create the first OLAP Services reporting application in production at Microsoft and is a co-author of “SQL Server 2000 Data Warehousing with Analysis Services” and “Transforming Healthcare through Information [Ed. Joan Ash] (2008)”. In addition to contributing to the SQLCAT Blog, SQL Server Best Practices, and SQLCAT.com, you can also review Denny's Space (http://denster.spaces.live.com). Denny specializes in developing solutions for Enterprise Data Warehousing, Analysis Services, and Data Mining; he also has focuses in the areas of Privacy and Healthcare.
|
sql, server, best practices, whitepapers, analysis services, data mining, olap, datawarehouse, datawarehousing, availability, clustering, capacity, collation, data types, data warehouse, database, design, index, mirroring, optimization, partitions, performance, precision, processing, querying, scalability, security, reporting services, integration services
|
|
