Data Mining NetFlow So What’s Next? Mark E Kane FloCon 2005 20 September 05

Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Download PDF Report

Upload
others
View
3
Download
0

Embed Size (px)

Citation preview

Page 1: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Data Mining NetFlowSo What’s Next?

Mark E KaneFloCon 200520 September 05

Page 2: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Objectives

Data Mining, very brieflyFrequency PatternsDiscoveriesRealizationsChanges Made

Data Mining

Data Mining – automated extraction of previously unknown data that is interesting and potentially useful.

Cost of Participating in Data Mining

Red Haring101010YESNO

Time Lost to Investigate and Clean

Up After Crime∞∞∞NOYES

-000NONO

Crime Prevented / Prosecuted101010YESYES

Result

Example SysAdmin

Hours

Example Investigator

Hours

Example Analyst Hours

Result of Data

MiningReality

Page 5: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Complexity of Mining NetFlow

Shear VolumeComplex Protocol AnalysisAmbiguous InterpretationsVery Smart Adversaries

Page 6: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Common Investigator Issues

Undermanned and overworkedVaried knowledge baseDoes not own networksNo direct reporting structure

Page 7: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Data Mining Techniques

Primary TechniquesRule and Tree InductionCharacterizationClassificationRegressionAssociationClustering

Other TechniquesDependency ModelingChange DetectionTrend AnalysisDeviation DetectionLink AnalysisPattern AnalysisSpatiotemporal Data MiningMining Path Traversal PatternsMining Sequential/Frequent Patterns

Uncertain Reasoning TechniquesFuzzy LogicNeural NetworksBayesian NetworksGenetic AlgorithmsRough Set Theory

Page 8: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Frequency Patterns

Mining Frequent Patterns in Data Streams in Multiple Time Granularities(Giennella, Han, Pei, Yan, and Yu)

Support Decision MakingPast Less Significant than PresentRecord ReductionTime Tilted Windows

Page 9: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Interpreting Time-Tilted Windows

DAYWindowTransition N Y N Y N Y N YSize 1 1 2 2 4 4 8 8

Monday 9Tuesday 15 9Wednesday 6 12Thursday 6 6 12Friday 12 6 12Saturday 16 12 6 12Sunday 6 14 9Monday 12 6 14 9Tuesday 15 9 14 9

0 1 2 3Day 1: 9 events

Day 2: 15 events (two buckets)

Day 3: 6 events (two buckets)

Day 4: 6 events (two buckets)

Day 5: 16 events (three buckets)

Day 6: 12 events (four buckets)

Page 10: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Presenting Frequency Patterns

Page 11: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Data Mining Discoveries

Failed email serversPreviously, unknown trusted relationshipsEncryption without authenticationPossible, but unproven intrusions

Page 12: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Data Mining Results

Frustrated InvestigatorsFrustrated AnalystsOne Very Frustrated Developer

Page 13: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Changes to Employ Data Mining

Establish common basis of understandingEstablish criteria for reporting

Geo-ResolutionTimelinessVolume

Establish reporting procedures

Page 14: Data Mining NetFlow...Cost of Participating in Data Mining NO YES 10 10 10 Red Haring Time Lost to Investigate and Clean Up After Crime YES NO ∞ ∞ ∞ NO NO 0 0 0 - Crime Prevented

Questions

Mark Kane

mkane @ ddktechgroup.com

s3-us-west-2.amazonaws.com · Web viewLow Back pain No Yes No Yes No Yes Osteoporosis No Yes No Yes No Yes Fractures No Yes No Yes No Yes Other: UCLA Activity Score: Please circle

Documents

Best Engineering College of Rajasthan | Top Engineering College.Bhamashah Card Aadhar Card For Office Use (Document Received) Yes No Yes / NO Yes No Yes / No Yes I No Yes / No Yes

Documents

Suggested ADULT PIF Changes · Web viewACLS ATLS PALS FCCS YES NO YES NO YES NO YES NO YES NO YES NO YES NO YES NO ACLS ATLS PALS FCCS YES NO YES NO YES NO YES NO YES NO YES NO YES

Documents

· GZ curve with heeling moment plot shown INTACT Yes No No No No No No 3D 3D 3D Yes No Yes Yes Yes YesGZ No No Yes No Yes No No Yes YesGZ Yes

Documents

· PDF filefor Saudi Aramco in 2012 (Valentine ... Blasting NO YES NO YES NO YES NO YES NO YES NO YES NO YES ... known to Raychem as tackifiers for

Documents

TAV Service Note Review Summation MH Care Coordination ... · 10) MH Oral Health Survey yes no yes no yes no yes no yes no 1) County of service yes no 2) yes no 7) 8) 3) yes no 4)

Documents

SF NEW PATIENT-FORMS 3-2017 - soulfocus.com · MRI, CT-Scan, Bone scan C] Yes a Yes C] Yes D Yes C] Yes C] Yes a Yes a Yes a Yes a Yes C] Yes a Yes C] No C) No No C] No No C] No [2

Documents

Drawing, Revision 1, 'Cover Sheet Sediment Control and … · 2012. 12. 4. · MnC2 RdD2 ReD SaB2 saC2 ShB2 ShC3 ShD3 SrE WoA Yes No No No Yes No No No No No Yes Yes Yes Yes Yes Yes

Documents

37-B 38-A 38-B 40.A 100 500 250 100 Yes Yes Yes Yes No No Yes Yes No Yes No Yes Yes Yes Yes Yes yes Yes Yes Yes Yes No No No Yes No Yes Yes Yes Smt Shabana Parveen Sh_ Sh_ Masque Masjid

Documents

Mail & Guardian · Kwa Dabeka CHC 13.1nanda CHC Providing Yes Yes Yes Yes Yes No No No No No Yes Yes Yes No No No No No Yes No No No No Yes Yes No No No Yes Yes Yes Yes No No No No

Documents

Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No No No No No No No No No No No No No No No No 9. Have you ever become ill, dizzy, or passed out while exercising in the heat? If

Documents

MDGS.TASE MDGS.Nasdaq Sept 2016 - medigus.com · Stomach acid rises ... MUSE™ EGS Torax yes yes yes yes yes yes 1 Mederi no no yes GI/Surgeon no no no no no no no 1 Surgeon

Documents

dentistryknoxvilletn.comdentistryknoxvilletn.com/download/i/mark_dl/u/1774949/...Taking oral contraceptives? Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes No No No No No no No No No blo

Documents

SCHOOL INSURANCE APPLICATION · Employees Volunteers Yes No Yes No Yes No Yes No Yes No Yes No Yes No years years Yes No. Page 5 of 12 Sexual Misconduct Liability If coverage is desired,

Documents

AGENCY/ORGANIZATION SPONSORSHIP FORM...REQUIRED FOR ALL PROGRAMS Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Assessment Fee (one time only $50.00) Bus Pass (First

Documents

Tranquility Vertical Stack (TSL) Series...A = ADA B A YES NO NO NO NO NO PSC PSC STANDARD LOW STATIC YES YES NO YES YES NO NO YES YES YES YES YES NO NO NO NO ECM CONSTANT VOLUME (DXM2

Documents

Distributed Object-Based Systemscrystal.uta.edu/~elmasri/os2/slides/tanenbaum/ch9.pdfMessaging No Yes Yes Events No Yes Yes Callbacks No Yes Yes Async. communication Yes Yes Yes Sync

Documents

FOOD WORKER ILLNESS FLOWCHART - … NO YES YES YES NO YES NO YES NO YES NO NO YES NO YES FOOD WORKER ILLNESS FLOWCHART Use this diagram to help you determine whether a food worker

Documents

· M. Irfan No No no Yes agreed Yes Yes No no 15000 Yes Ashraf Ali Yes yes yes yes yes yes Yes Yes yes 32786 yes Feed Well Yes Yes Yes Yes agreed Yes Yes Yes Yes 26000 Yes Al-Wahid

Documents

Laser Spa Grouplaserspagroup.com/LSG-Insurance-Application.pdf · Sell Products under own label? Hair Cutting / Coloring Cl YES NO YES YES a NO NO YES YES YES YES YES YES a NO NO

Documents

TELL US ABOUT YOUR CHILD€¦ · Discolored Teeth Bad Breath Mouth Trauma/Broken Tooth Teeth Grinding Sensitivity to Hot/Cold Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes

Documents

Theory manual ENG 2011 - DeepEx · Caquot-Kerisel Tabulated No - - - Yes Yes Yes No Caquot-Kerisel Tabulated No - - - Yes Yes Yes No Lancellota No - - - Yes Yes Yes Yes Notes: 1

Documents

EDIC@ Specifications - medicacorp.com · Yes Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No CALIBRATE NOW? ANALYZE BLOOD? DAILY CLEANER?/ STANDBY MODE? SECOND

Documents

APPLICATION FOR PHYSICAL THERAPY - ksbha.org(p) Yes No (n) Yes No (s) Yes No (t) Yes No (u) Yes No (q) Yes No (r) Yes No Have you ever been charged with a crime, indicted, convicted

Documents

MDA Health History - eng - mkedentalarts.com · Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Dental History No No No No No No No No No No 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. Do

Documents

· 2014-12-05 · Blood Preswe Yes No Yes No Yes Yee Yes Yes Yes Yes Yes Yes Yes NO NO No NO No No No No No Hena Asthma AIDS or HtV+ Ulcer Thyoid Osease Kidney Disease Yes No Yes

Documents

Table of Contents - download.ivanti.com · Mantis 2.1.0+ SOAP Yes Yes Yes Yes VSTS / TFS 2017+ Basic Yes Yes Yes Yes SalesForce OAuth Yes Yes No No HP Quality Center Cookies No No

Documents

docs/Ibstock... · Web viewyes / no. yes / no. yes / no. yes / no. yes / no. yes / no. yes / no. yes / no. yes / no. yes / no. yes / no. yes / no. yes / no. yes / no

Documents

Office Policy - lpwellness.com · Tonsillitis Yes No Bleeding Disorders Yes No Gout Yes No Pacemaker Yes No Tuberculosis Yes No Breast Lump/Cyst Yes No Headaches Yes No Parkinson’s

Documents

Section 3 – General Liability and Excess Complete Sections 3 thru … · 1 day ago · Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No Yes No . Section 4 B. - Property

Documents