Embed Size (px)
Citation preview
ThreatConnect® App For Splunk Enterprise User Documentation 23 November 2014
1 ThreatConnect, Inc.
Table of Contents
Overview
Key Features
Getting Started
Install
ThreatConnect API User Creation
ThreatConnect App for Splunk
App Setup and Configuration
Setting Up Alerts
Basic Alert Scenario
ThreatConnect Dashboard
Indicator Dashboard
Threat Lookup
Reports
DB Data
Search
Workflow - Event Actions
Workflow - Field Actions
Inputlookup with ThreatConnect Indicator Data
Enriched Events with ThreatConnect Indicator Data
ThreatConnect Logs
Indicator Logs
Group Logs
Tag Logs
2 ThreatConnect, Inc.
Overview The ThreatConnect® App for Splunk Enterprise gives Splunk users the ability to leverage customizable threat intelligence integrated into Splunk from their ThreatConnect accounts. ThreatConnect provides the ability to aggregate threat intelligence from multiple sources (open source, commercial, communities, and internally created), analyze and track identified adversary infrastructure and capabilities, and put that refined knowledge to work in Splunk identifying threats targeting your organization.
Key Features
● Transparent threat intelligence context from multiple sources (open source, commercial, communities, and
from internal research) applied to triggered Splunk Alerts.
● Automatic updating of threat intelligence indicators from ThreatConnect Cloud, Private Cloud, or On-
Premises platforms.
● Prioritize matched events based on criticality and confidence scores, relationships to known Threat types
and groups, past incidents, and tags.
● High level dashboards showing details on matches from indicators by indicator type, criticality, associations,
and many other features.
Getting Started The download of the ThreatConnect App for Splunk Enterprise includes a sample set of data for demo purposes. You may use this to preview the basic functionality of the application without a ThreatConnect API account. If you wish to trial the ThreatConnect App for Splunk with a live connection to the latest customizable threat intelligence data, please register for a free no-obligation trial of ThreatConnect here. Once your Organization has been licensed for API access, you will need to create an API User within your Organization prior to Splunk interfacing with the ThreatConnect API. For detailed steps on creating an API user, please see ThreatConnect API Account Creation.
Install There are a few options for installing the ThreatConnect App for Splunk Enterprise. For more information in installing Splunk Apps, reference the Splunk documentation located here.
3 ThreatConnect, Inc.
Install Directly from Splunk Apps:
1. From the Splunk web interface, go to the Apps menu in the upper left-hand corner, and select "Manage Apps". 2. Click the "Find more apps online" button. 3. Browse for, or Search for the " ThreatConnect App for Splunk Enterprise " app. 4. Click on the "Install free" button. Note that you may be required to login with your Splunk Apps credentials.
Download and install using the Splunk Web interface
1. Download the app from Splunk Apps. 2. From the Splunk web interface, go to the Apps menu in the upper left-hand corner, and select "Manage Apps". 3. Click on the "Install app from file" button. 4. Click on the "Choose File" button to browse for the file you downloaded. 5. Click "Upload" to install the app.
Download and install to the Splunk file system
1. Download the app from Splunk Apps. 2. Copy the downloaded file to the $SPLUNK_HOME/etc/apps directory 3. Decompress the file using the tool of your choice. 4. Restart Splunk
ThreatConnect API User Creation A ThreatConnect API User is created from within the ThreatConnect Web application for the instance you are using, for the ThreatConnect Public Cloud edition this is: https://app.threatconnect.com. To create an API User, you must login with an Organization Administrator account. After logging in, under the User Name drop down on the menu bar, click the Org Settings link. On the Organization Settings page, in the upper right corner within the Membership tab, click the “Create API User” button. Fill in the information requested and copy the Access ID and Secret Key to a safe location. It is critical to save the Secret Key as this will not be available again. If it is lost, the existing API User will need to be deleted and a new API User will need to be created. Do not share the Access ID or Secret Key as these are your credentials and will be used to track API usage limits. The following fields are required:
4 ThreatConnect, Inc.
Parameter Description
First Name The First Name of the user that will appear in Posts, data modifications and data creation within ThreatConnect. *
Last Name The Last Name of the user that will appear in Posts, data modifications and data creation within ThreatConnect. *
Pseudonym The Pseudonym of the user that will appear in Posts, data modifications and data creation within ThreatConnect. *
* Currently the ThreatConnect API is read-only. These fields will be used at a later date when write functionality is enabled.
5 ThreatConnect, Inc.
ThreatConnect App for Splunk
App Setup and Configuration URL: https://[Splunk Hostname]/en-US/manager/ThreatConnectApp/apps/local/ThreatConnectApp/setup?action=edit
After installing the ThreatConnect App for Splunk Enterprise, the application setup must be completed before accessing the app. To properly configure the App, fill in each of the form text boxes with the appropriate data from the API User Creation section.
6 ThreatConnect, Inc.
Default Organization: Your Organization as defined during account creation within the ThreatConnect application.
API Base URL: The ThreatConnect Public Cloud API can be accessed at https://api.threatconnect.com. If you have a Private Cloud or On-Premises edition of ThreatConnect, the URL for your instance was provided to you during your initial setup and install.
API Access ID: The API Access ID corresponds with your ThreatConnect API user account’s Access ID.
API Secret Key: The API Secret Key corresponds with your ThreatConnect API user account’s key accessible during account creation within your ThreatConnect organization.
7 ThreatConnect, Inc.
API Max Results: The maximum value of results that can be returned in one query without pagination. The maximum value currently supported with the ThreatConnect API for returned results without pagination is 500. Unless there is a special requirement on your network, it is recommended this value be left at 500.
TC Indicator Download: Select this checkbox to enable the download of all indicators from all communities that the API key provides membership. Without this enabled, no indicator data will be downloaded and alerting will use the last downloaded indicator set. This download can be run manually by selecting “Searches, reports, and alerts” from the Settings menu. Then select “run” from the Action column for the “TC-Indicator-Download”. The saved search page can also be accessed from https://[Splunk Hostname]/en-US/app/ThreatConnectApp/saved/searches.
NOTE: If you are using the app without a valid ThreatConnect API account, leave this box unchecked.
Modify Download Scheduler: The schedule for the Indicator download can be defined here. The recommended download period is once every 24 hours. More information on CRON settings can be found at http://en.wikipedia.org/wiki/Cron.
After the setup in complete the ThreatConnect App for Splunk will be accessible. When accessing the App the default dashboard will show no results, which is expected. This dashboard will not show populated results until matched data is available. Alerts must be created to specify which log and event data within Splunk to match against.
Setting Up Alerts Any search that returns a set of indicators can be saved as an alert that automatically searches for matching indicators from ThreatConnect. The search can return any single indicator type or multiple indicator types that are supported by the ThreatConnect API. There are two additional steps that are required to implement these automated ThreatConnect queries once the search has been created.
Step One: Use the Splunk built-in “rename” command to rename the indicator field to “indicator” (e.g. sourcetype=cisco:asa action=drop | stats count by src_ip | where count > 500 | rename src_ip as indicator).
Step Two: Once the search command is returning valid indicator results with a field name of “indicator” the results need to be sent to the ThreatConnect script (e.g. sourcetype=cisco:asa action=drop | stats count by src_ip | where count > 500 | rename src_ip as indicator | script tc_alert).
It is recommended to run the search over the past hour of logs and set the alert to run every hour. If any matches are found, a log entry will be created in the ThreatConnect App log and can be viewed from the App’s ThreatConnect Dashboard or by executing a search on the ThreatConnect sourcetype (e.g. sourcetype=threatconnect match=True).
All non-matching calls to the ThreatConnect script will also create a log entry and can be viewed by searching the ThreatConnect sourcetype (e.g. sourcetype=threatconnect match=False). The non-matching events are not displayed in the Dashboard.
8 ThreatConnect, Inc.
Basic Alert Scenario The company has a requirement to monitor Internet traffic sourced from China coming into the network. The search could be as simple as “sourcetype=cisco:asa | iplocation src_ip | search Country=China | rename src_ip as indicator”. However, this could create a lot of results and trigger too many lookups. We can filter these results down to all traffic sourced from China that is not blocked by the firewall by adding an additional filter. The new search would be “sourcetype=cisco:asa action=permit | iplocation src_ip | search Country=China | rename src_ip as indicator”. This search will now return fewer results. The search can be filtered further to exclude duplicate IP addresses by using the stats command to get the count of events by src_ip. To limit the results even more a threshold can be added to the search using the where command. The new search may look something like “sourcetype=cisco:asa action=permit | iplocation src_ip | search Country=China | stats count by src_ip | where count > 50 | rename src_ip as indicator”.
Now that the search results have been filtered to a comfortable level they can be passed to the ThreatConnect script. Ensure that the indicator field has been renamed to “indicator” and that “| script tc_alert“ has been appended to the search. Then click on the “Save As” link on the top right of the search screen. From the dropdown menu select “Alert”.
The alert title can be any value, however it is recommended to add a meaningful name for ease of recognition. Adding a description will also help others understand the purpose of the search.
Ensure that the search time period is equal to the frequency of the Alert so that results are not missed.
9 ThreatConnect, Inc.
Finalize the Alert configuration and click Save.
10 ThreatConnect, Inc.
This is a basic sample of creating a search to run as an alert. There are unlimited combinations of searches that can be created to return indicators that are meaningful to threat analysis. These searches can then be configured as an alert to automatically match indicators from ThreatConnect. All that is required is knowledge of your security logs and an understanding of what type of events can return useful data for threat analysis.
ThreatConnect Dashboard URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/tc_view_main
The ThreatConnect Dashboard provides an overview of matches between events in Splunk and indicator data in ThreatConnect.
The first row of Single Valued Results provides a count of matched indicators. These indicators are separated by Indicator Type. You can use the time picker above the first row to select what time frame to view relevant matching hits from ThreatConnect.
11 ThreatConnect, Inc.
The DrillDown action for these results will take you to the Splunk search that generated the count. Note that the count de-duplicates the indicator by each API lookup so the number of events might be different depending on how many communities the indicator belongs. See the Search Section for more information on viewing the generated logs from matched indicators.
The second row provides a graphical view of Matched Indicator by Description, Rating, and Type. This view provides an oversight of indicator activity in Splunk.
The view at the bottom displays the latest matched indicators in a paginated table. This table has a built-in form that allows dynamic filtering on indicator data. Note that a matching event may display several times if the matched indicator has several owners in ThreatConnect. In cases such as these, a row will be displayed for each owner of the indicator.
Each column is described below:
_time: The _time field is stored internally in UTC format. It is translated to human-readable Unix time format when Splunk Enterprise renders the search results (the very last step of search time event processing).
Indicator: Lists the indicator that matched between local logs and ThreatConnect. This value is a hyperlink that will open a page to the indicator’s detail page on the ThreatConnect website.
Trigger: The trigger field indicates whether a match event was caused by an alert or a manual search against the ThreatConnect API. Clicking on the value will run a search in Splunk to the relevant logs generated by the ThreatConnect App and for alert type triggers, the matching original logs or events indexed by Splunk.
Rating: The rating field lists the criticality rating assigned by the indicator’s owner within ThreatConnect. This is a scale of 0 to 5 with 5 being the most critical. The ratings are color highlighted by their value (0-2.99 no highlight, 3-4 amber and >4 red).
12 ThreatConnect, Inc.
Confidence: The confidence field lists the confidence rating assigned by the indicator’s owner within ThreatConnect. This is a percentage scale.
Owner: The owner field displays the owner of the indicator within ThreatConnect. This is typically a particular source, community, or the organization’s own private data. The value of this field is hyperlinked to the owner’s dashboard page within ThreatConnect.
Tags: The tags field will display any tags associated with the matched indicator by the owner. If multiple owners exist for a matched indicator, only tags created by the owner listed in the same row will be displayed in tag column.
Group (Types): The group field will display any groups associated with the matched indicator by the owner as well as the group type (e.g. Incidents, Emails, Threats, Emails, Adversaries, or Signatures). If multiple owners exist for a matched indicator, only groups created by the owner listed in the same row will be displayed in the “Group (Types)” column.
Indicator Dashboard URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/tc_view_indicators
The Indicator Dashboard provides an additional view of the matched indicator data. This dashboard focuses on the groupings of the matched indicators themselves and provides some statistics on the most recent indicators available from ThreatConnect.
The first row of this graphical view displays Matched Indicators by Owners (e.g. Source, Community, Organization), Group (e.g. specific Incidents, Threats, Emails, Signatures, or Adversaries), and Tags.
13 ThreatConnect, Inc.
The second row displays additional paginated tables for matched indicators. From left to right these are the top matched indicators with a count of times observed for each indicator, the top matched group names, and top matched tags.
The third row provides paginated tables for the last 20 added indicators and the last 20 modified indicators. Toggling back and forth between last 20 added and modified indicator tables is enabled with a quick link switcher at the top of the tables.
Note: The lastModified date is auto populated with the date added value on Indicator creation so these fields could contain the same value.
14 ThreatConnect, Inc.
Threat Lookup URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/tc_view_manual_lookup
This page allows manual lookup of indicators against the ThreatConnect API. The indicator type will be automatically detected. This manual lookup will be logged to the App logs and will be displayed on the App Dashboard.
Reports URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/reports
A few canned reports are created for monitoring ThreatConnect alert queries and how many of those queries hit the ThreatConnect API. User defined custom reports can be added for more detailed views into the ThreatConnect data.
15 ThreatConnect, Inc.
DB Data The DB Data pulldown menu provides additional pages for each ThreatConnect indicator type independent of matches to events or logs within Splunk, allowing users to view statistics for each. The pages are formatted identically.
One page exists for each indicator type:
Address Indicators DB Data Page URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/tc_view_indicator_addresses
Email Addresses DB Data Page URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/tc_view_indicator_emailaddresses
File Indicators DB Data Page URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/tc_view_indicator_files
Host Indicators DB Data Page URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/tc_view_indicator_hosts
URL Indicators DB Data Page URL: https://[Splunk Hostname] /en-US/app/ThreatConnectApp/tc_view_indicator_urls
16 ThreatConnect, Inc.
The first row displays graphical representations for the total number of indicators from ThreatConnect of the specified type. There are two charts for Indicator type by owner and by rating.
The second row contains two tables to display the last 10 created and updated indicators.
The final row on the page displays a paginated table of all the indicators of that type pulled from ThreatConnect. For Address Indicators a quick link switcher is available to also view a country chart and country map showing the geographic distribution of the IP addresses from ThreatConnect.
17 ThreatConnect, Inc.
Search URL: https://[Splunk Hostname]/en-US/app/ThreatConnectApp/search
Workflow - Event Actions Using the Search page while in the ThreatConnect App provides additional features for threat analysis. The built-in Splunk Event Actions feature will display a link for any indicator field that follows the Common Information Model (CIM) standard naming convention. By selecting this link, the indicator displayed in the link will be queried using the ThreatConnect API and any matching result will be displayed in a results table.
18 ThreatConnect, Inc.
19 ThreatConnect, Inc.
20 ThreatConnect, Inc.
Workflow - Field Actions If the results fields do not follow the CIM naming convention, an indicator can still be queried in the ThreatConnect API by using the Field Actions menu. By clicking/selecting this link, the indicator displayed in the link will be queried using the ThreatConnect API and any matching result will be displayed in a results table.
21 ThreatConnect, Inc.
22 ThreatConnect, Inc.
Inputlookup with ThreatConnect Indicator Data ThreatConnect indicator data is always available to use in custom searches. The data can be accessed using the Splunk built-in inputlookup command (e.g. “| inputlookup addresses_indicators.csv”). There is a csv file available for each indicator type.
Enriched Events with ThreatConnect Indicator Data This is a simple example to append the data in ThreatConnect CSV files to your existing logs. The search would look something like “sourcetype=cisco:asa action=drop | join src_ip [ | inputlookup addresses_indicators.csv | rename * as tc_* | rename tc_ip as src_ip ]”. This search would find any firewall traffic that had an action of “drop” and join the ThreatConnect Data to the event on a matching src_ip field. Only events that matched on both sides of the join will be displayed. The ThreatConnect data would be prepended with “tc_” so that it can be easily distinguished from the firewall event data. Now that the firewall data is enriched with ThreatConnect data, further filtering and analysis can be performed on the enriched events.
23 ThreatConnect, Inc.
Note that the results will only return indicators data that were downloaded during the last indicator pull and will not query the ThreatConnect API for any additional information.
ThreatConnect Logs The ThreatConnect App logs can be searched just like any other log in Splunk. The App logs are formatted in JavaScript Object Notation (JSON) and are easily readable by humans and machines.
Indicator Logs All indicators that are passed to the ThreatConnect script are logged. The image below displays a matching result in it native collapsed form.
24 ThreatConnect, Inc.
By clicking on the “+” symbol in the log event you can expand the event to display additional data. The image below displays a fully expanded matching indicator result in the ThreatConnect App log. All the indicator data is nested in the “indicator” section of the event. All the search information, including the search string that created this log entry, is nested in the “search” section of the event. A key field in this data is search.method which indicates whether the search was performed manually using the “ThreatConnect Manual Lookup”, Workflow Action, or by an alert.
Each field is described from the log below:
app: this field notates the ThreatConnect App as the source of the log.
indicator: this field is nested with data on the matched indicator from the ThreatConnect API.
indicator.confidence: the owner specific confidence value on the indicator provided from the ThreatConnect API.
indicator.dateAdded: the date the indicator was added to the owner within ThreatConnect.
indicator.description: the owner specific default description attribute of the indicator from ThreatConnect.
indicator.id: the indicator’s id value from ThreatConnect.
indicator.indicator: the actual indicator value from ThreatConnect.
indicator.lastModified: the time the indicator was last modified within ThreatConnect.
indicator.owner: the owner field is nested with data on the indicator’s owner from ThreatConnect.
indicator.owner.id: the owner’s id value from ThreatConnect.
indicator.owner.name: the name of the indicator’s owner within ThreatConnect.
indicator.owner.type: the type of owner.
indicator.rating: the owner specific criticality rating of the indicator .
25 ThreatConnect, Inc.
indicator.source: the owner specific default source attribute value for the indicator from ThreatConnect.
indicator.email: this field is present if the matched indicator is an email address. Its value will be the email address indicator.
indicator.hostname: this field is present if the matched indicator is a host name. Its value will be the host indicator.
indicator.ip: this field is present if the matched indicator is an IP address. Its value will be the IP address indicator.
indicator.md5: this field is present if the matched indicator is a MD5. Its value will be the MD5 hash indicator.
indicator.sha1: this field is present if the matched indicator is a SHA1. Its value will be the SHA1 hash indicator.
indicator.sha256: this field is present if the matched indicator is a SHA256. Its value will be the SHA256 hash indicator.
indicator.text: this field is present if the matched indicator is a URL. Its value will be the URL indicator.
indicator.type: the type of indicator that was automatically determined by the ThreatConnect script
indicator.weblink: a unique url link to the indicator’s Details Page within the ThreatConnect platform.
match: A value of “True” indicates that the indicator that was searched matched a known indicator in ThreatConnect. A value of “False” indicates that the indicator that was searched did not match a known indicator in ThreatConnect.
search: this field is nested with data relevant to the search made within the ThreatConnect App.
search.indicator: this field will be identical to the indicator.indicator on event where match=True. Events that have match=False will not have an indicator section.
search.method: A value of “alert” in this field indicates the log event was triggered by an alert. A value of “manual” indicates that the log event was generated from a ThreatConnect Manual Lookup or a Workflow Action.
search.owners: the ThreatConnect owners that were searched for the value in search.indicator.
search.string: the search string that created this event.
search.type: the type of search performed in the ThreatConnect App. The values can be indicators, groups, or tags.
timestamp: the timestamp for this logged event.
uuid: the universally unique id value for this logged event. The uuid field can be used to join indicator events with matching tag and group events for the same search.
26 ThreatConnect, Inc.
The following image displays a fully expanded indicator event that did not match during the ThreatConnect script execution. The key field to note here is “match: False”. This information is useful in tuning the alert to filter out
27 ThreatConnect, Inc.
additional indicators. It also provides a view into what is defined as a relevant indicator to trigger an alert.
Group Logs Each time a matching indicator is found the ThreatConnect script also performs a query to find any matching group related to the matching indicator. If there are any matching groups the group data will be written to the App log file. Below is a fully expanded log entry for a group. This group can be associated with the matching indicator log entry using the uuid field.
Each field unique to the group log is described below:
group: this field is nested with data relevant to a group associated to a matched indicator.
group.dateAdded: date an owner specific group was added to ThreatConnect.
group.id: the unique id of the group within ThreatConnect.
group.name: the name of the group within ThreatConnect.
group.ownerName: the name of the owner of the group within ThreatConnect.
group.type: the type of group.
group.weblink: a url link to the group’s Details page within ThreatConnect.
28 ThreatConnect, Inc.
Tag Logs Each time a matching indicator is found, the ThreatConnect App also performs a query to find any matching tags related to the matching indicator. If there are any matching tags, the tag data will be written to the App log file. Below is a fully expanded log entry for a tag. This tag can be associated with the matching indicator log entry using the uuid field.
Each field unique to the group log is described below:
tag: this field is nested with data relevant to a tag associated to a matched indicator.
tag.name: the name of the tag within ThreatConnect.
tag.weblink: a url link to the tag’s Details page within ThreatConnect.
29 ThreatConnect, Inc.
30 ThreatConnect, Inc.
