Suspicious Connects

Access the analyst view for Suspicious Connects http://"server-ip":8889/files/ui/flow/suspicious.html. Select the date that you want to review (defaults to current date).

Your view should look similar to the one below:

Suspicious Connects Web Page contains 4 frames with different functions and information:

• Suspicious
• Network View
• Scoring
• Details

The Suspicious frame

Located in the top left corner of the Suspicious Connects Web Page, this frame presents the Top 250 Suspicious Connections in a table format based on Machine Learning (ML) output. These are the columns depicted in this table:

• Rank - ML output rank
• Time - Time received field for Netflow record
• Source IP - Netflow Record Source IP Address
• Destination IP - Netflow Record Destination IP Address
• Source Port - Netflow Record TCP/UDP Source Port Number
• Destination Port - Netflow Record TCP/UDP Destination Port Number
• Protocol - Text format for Protocol contained within Netflow Record (Ex. TCP/UDP)
• Input Packets - Reported Input Packets for the Netflow Record
• Input Bytes - Reported Input Bytes for the Netflow Record
• Output Packets - Reported Output Packets for the Netflow Record
• Output Bytes - Reported Output Bytes for the Netflow Record

Additional functionality in Suspicious frame

1. By selecting a specific row within the Suspicious frame, the connection in the Network View will be highlighted.
   
   
2. In addition, by performing this row selection the Details Frame presents all the Netflow records in between Source & Destination IP Addresses that happened in the same minute as the Suspicious Record selected
   
   
3. Next to a Source/Destination IP Addresses, a shield icon might be present. This icon denotes any reputation services value context added as part of the Operational Analytics component. By rolling over you can see the IP Address Reputation result
   
   
4. An additional icon next to the IP addresses within the Suspicious frame is the globe icon. This icon denotes Geo-localization information context added as part of the Operational Analytics component. By rolling over you can see the additional information
   
   
   
   

The Network View frame

Located at the top right corner of the Suspicious Connects Web Page. It is a graphical representation of the Suspicious records relationships.

If context has been added, Internal IP Addresses will be presented as diamonds and External IP Addresses as circles.

Additional functionality in Network View frame

1. As soon as you move your mouse over a node, a dialog shows IP address information of that particular node.

   

2. A primary mouse click over one of the nodes will bring a chord diagram into the Details frame.

   The chord diagram is a graphical representation of the connections between the selected node and other nodes within Suspicious Connects records, providing number of Bytes From & To. You can move your mouse over an IP to get additional information. In addition, drag the chord graph to change its orientation.

   

3. A secondary mouse click uses the node information in order to apply an IP filter to the Suspicious Web Page.

   

Scoring Frame

This frame contains a section where the analyst can score IP Addresses and Ports with different values. In order to assign a risk to a specific connection, select it using a combination of all the combo boxes, select the correct risk rating (1=High risk, 2 = Medium/Potential risk, 3 = Low/Accepted risk) and click Score button. Selecting a value from each list will narrow down the coincidences, therefore if the analyst wishes to score all connections with one same relevant attribute (i.e. src port 80), then select only the combo boxes that are relevant and leave the rest at the first row at the top.

The Score button

When the Analyst clicks on the Score button, the action will find all coincidences exactly matching the selected values and update their score to the rating selected in the radio button list.

The Save button

Analysts must use Save button in order to store the scored connections. After you click it, the rest of the frames in the page will be refreshed and the connections that you already scored will disappear on the suspicious connects page, including from the lists. At the same time, the scored connections will be made available for the ML. The following values will be obtained from the spot.conf file:

• LUSER

The Quick IP Scoring box

This box allows the Analyst to enter an IP Address and scored using the "Score" and "Save" buttons using the same process depicted above.

Threat Investigation

Access the analyst view for suspicious connects http://"server-ip":8889/files/ui/flow/suspicious.html.

Select the date that you want to review.

Your screen should now look like this:

The analyst must score the suspicious connections before moving into Threat Investigation View, please refer to Suspicious Connects Analyst View walk-through.

Select Flows > Threat Investigation from apache spot (incubating) Menu.

Threat Investigation Web Page will be opened, loading the embedded IPython notebook.

Expanded search

You can select any IP from the list and click Search to view specific details about it. A query to the flow table will be executed looking into the raw data initially collected to find all communication between this and any other IP Addresses during the day, collecting additional information, such as:

• max & avg number of bytes sent/received
• max & avg number of packets sent/received
• destination port
• source port
• first & last connection time
• count of connections

The full output of this query is stored into the flow_threat_investigation table.

Based on the results from this query, a table containing the results will be displayed with the following information:

• Top 'n' IP's per number of connections.
• Top 'n' IP's per bytes transferred.
• The number of results stored in the dictionaries (n) can be set by updating the value of the top_results variable.

Save Comments

In addition, a web form is displayed under the title of 'Threat summary', where the analyst can enter a Title & Description on the kind of attack/behavior described by the particular IP address that is under investigation.

Click on the Save button after entering the data to write it into the flow_storyboard table.

At the same time, the charts for the storyboard will be created. Depending on the existence of the geolocation database, the following graphs will be generated:

Map view - create a globe map indicating the trajectory of the connections based on their geolocation.
Output: globe_[threat_ip] .json

Impact analysis - This will represent the number of inbound, outbound and twoway connections found.
Output: stats-[threat_ip] .json

Dendrogram - This represents all different IP's that have connected to the IP under investigation, this will be displayed in the Storyboard under the Incident Progression panel as a dendrogram. If no network context file is included, the dendrogram will only be 1 level deep, but if a network context file is included, additional levels will be added to the dendrogram to break down the threat activity.
Output: dendro-[threat_ip] .json

Timeline - This represents additional details on the IP under investigation and its connections grouping them by time; so the result will be a graph showing the number of connections occurring in a customizable timeframe.
Output: flow_timeline.tsv (Impala table)

Executive threat briefing - Here the comments for the IP are displayed as a menu under the 'Executive Threat Briefing' panel.
Output: flow_storyboard (Impala table)

Once you have saved comments on any suspicious IP, you can continue to the Storyboard to check the results.

Storyboard

1. Select the option Flow > Storyboard from Apache Spot (incubating) Menu.

   

2. Your view should look something like this, depending on the IP's you have analyzed on the Threat Analysis for that day. You can select a different date from the calendar.

   

3. Review the results:

   Executive Threat Briefing

   Executive Threat Briefing lists all the incident titles you entered at the Threat Investigation notebook. You can click on any title and the additional information will be displayed.

   

   Clicking on a threat from the list will also update the additional frames.

   Incident Progression

   Frame located in the top right of the Storyboard Web page

   

   Incident Progression displays a tree graph (dendrogram) detailing the type of connections that conform the activity related to the threat.

   When network context is available, this graph will present an extra level to break down each type of connection into detailed context.

   Impact Analysis

   

   Impact Analysis displays a horizontal bar graph representing the number of inbound, outbound and two-way connections found related to the threat. Clicking any bar in the graph, will break down that information into its context.

   Map View | Globe

   

   Map View Globe will only be created if you have a geolocation database. This is intended to represent on a global scale the communication detected, using the geolocation data of each IP to print lines on the map showing the flow of the data.

   Timeline

   

   Timeline is created using the resulting connections found during the Threat Investigation process. It will display 'clusters' of inbound connections to the IP, grouped by time; showing an overall idea of the times during the day with the most activity. You can zoom in or out into the graphs timeline using your mouse scroll.

Ingest Summary

1. Load the Ingest Summary page by going to http://"server-ip":8889/files/index_ingest.html or using the drop down menu.

   

2. Select a start date, end date and click the reload button to load ingest data. Ingest summary will default to last 7 seven days. Your view should now look like this:

   

3. Ingest Summary presents the Flows ingestion timeline, showing the total flows for a particular period of time.

   • Analyst can zoom in/out on the graph.

   By default, the ingested data for netflow will be displayed. If you want to review a different pipeline, you can change the selection in the dropdown list at the top of the page.

   

Suspicious DNS

Open the analyst view for Suspicious DNS: http://"server-ip":8889/files/ui/dns/suspicious.html. Select the date that you want to review (defaults to current date).

Your screen should now look like this:

Suspicious Connects Web Page contains 4 frames with different functions and information:

2. Suspicious
3. Network View
4. Scoring

The Suspicious Connections
Located at the top left of the Web page, this frame shows the top 250 suspicious DNS from the Machine Learning (ML) output.

1. By moving the mouse over a suspicious DNS, you will highlight the entire row as well as a blur effect that allows you to quickly identify current connection within the Network View frame.
   
   
2. Shield icon. Represents the output for any Reputation Services results that has been enabled, user can mouse over in order to obtain additional information. The icon will change its color depending upon the results from specific reputation services.
   
   
3. By selecting on a Suspicious DNS record, you will highlight current row as well as the node from Network View frame. In addition Details frame will be populated with additional communications directed to the same DNS record.
   
   

The Network View Frame
Located at the top right corner, Network View is a graphic representation of the "Suspicious DNS".

1. As soon as you move your mouse over a node, a dialog shows up providing additional information.
   
   
2. Diamonds represents DNS records and circles represents IP addresses communicating to the respective DNS record
   
   
3. A primary mouse click in an IP Address (circle) will bring a diagram within Details frame, providing all the Domain Name records queried by that particular IP Address
   
   
4. A secondary mouse click uses the node information to filter suspicious data.
   
   

The Details Frame
Located at the bottom right corner of the Web page. It provides additional information for the selected connection.

Detail View frame has two modes:

• Table details (when you select a record in the Suspicious frame).
• Dendrogram diagram (when you select an IP address in the Network View frame)

Scoring Frame
The main function in this frame is to allow the Analyst to score IP Addresses and DNS records with different values. In order to assign a risk to a specific connection, select it using a combination of all the combo boxes, select the correct risk rating (1=High risk, 2 = Medium/Potential risk, 3 = Low/Accepted risk) and click Score button. Selecting a value from each list will narrow down the coincidences, therefore if the analyst wishes to score all connections with one same relevant attribute (i.e. ip address 10.1.1.1), then select only the combo boxes that are relevant and leave the rest at the first row at the top.

Score button

Pressing the 'Score' button will find all exact matches of the selected threat (Client IP or Query) in the dns_scores table to set their severity value to the one selected.

Selecting values from both the "Client IP" and "Query" lists to score them together, will update every matching threat individually with the same rating value, but not necessarily as a Client_IP-Query pair.

You can score a large set of similar or coincident queries by entering a keyword in the "Quick Scoring" text field and then select a severity value from the radiobutton list. The value entered here will only search for matches on the dns_qry_name name column. "Quick Scoring" text field has precedence over any selection made on the lists.

The Save button

Analysts must use the Save button in order to save the scored records into the database, in the dns_threat_investigation table. After you clicking it, the rest of the frames in the page will be refreshed and the connections that you already scored will disappear on the suspicious connects page. At the same time, the scored connections will be made available for the ML to use as feedback. The following values will be obtained from the .conf file:

• LUSER

DNS Threat Investigation

Access the analyst view for DNS Suspicious Connects. Select the date that you want to review.

Your view should now look like this:

The analyst must previously score the suspicious connections before moving into Threat Investigation View, please refer to DNS Suspicious Connects Analyst View walk-through.

Select DNS > Threat Investigation from Apache Spot (incubating) Menu.

Threat Investigation Web Page will be opened, loading the embedded IPython notebook. A list with all IPs and DNS Names scored as High risk will be presented

Expanded Search

Select any value from the list and press the "Search" button. The system will execute a query to the dns table, looking into the raw data initially collected to find additional activity of the selected IP or DNS Name according to the following criteria:

Expanded Search for a particular Domain Name

The query results will provide the different unique IP Addresses list that have queried this particular Domain, the list will be sorted by the quantity of connections.

Expanded Search for a particular IP

The expanded search will provide the different unique Domains list that this particular IP queried in one day, they will be sorted by the quantity of connections made to each specific Domain Name.

The full output of this query is stored into the flow_threat_investigation table. Based on the results from this query, a table containing the results will be displayed with the following information. The quantity of results displayed on screen can be set by modifying the top_results variable.

Save comments.

In addition, a web form is displayed under the title of 'Threat summary', where the analyst can enter a Title & Description on the kind of attack/behavior described by the particular IP or DNS query name address that is under investigation.

Click on the "Save" button after entering the data to write it into the dns_storyboard table. At the same time, the charts for the storyboard will be created.

Continue to the Storyboard.

Once you have saved comments on any suspicious IP or domain, you can continue to the Storyboard to check the results.

DNS Storyboard

Walk-through

1. Select the option DNS > Storyboard from Apache Spot (incubating) Menu.

   

2. Your view should look something like this, pending on how many threats you have analyzed and commented on the Threat Analysis for that day. You can select a different date from the calendar.

   

Executive Threat Briefing

Executive Threat Briefing frame lists all the incident titles you entered at the Threat Investigation notebook. You can click on any title and view the additional comments at the bottom area of the panel.

Incident progression frame is located on the right side of the Web page.

This will display a tree graph (dendrogram) detailing the type of connections that conform the activity related to the threat.

Suspicious Proxy

1. Open the analyst view for Suspicious Proxy: http://"server-ip":8889/files/ui/proxy/suspicious.html. Select the date that you want to review (defaults to current date).

   Your screen should now look like this:

   
   
   

   Suspicious Connects Web Page contains 4 frames with different functions and information:

   • Suspicious
   • Network View
   • Scoring
   • Details

2. Suspicious Frame
   Located at the top left of the Web page, this frame shows the top 250 Suspicious Proxy connections from the Machine Learning (ML) output.

   1. By moving the mouse over a suspicious Proxy record, you will highlight the entire row as well as a blur effect that allows you to quickly identify current connection within the Network View frame.
      
      
   2. The Shield icon. Represents the output for any Reputation Services results that has been enabled, user can mouse over in order to obtain additional information. The icon will change its color depending upon the results from the Reputation Service.
      
      
   3. The List icon. When the user mouse over this icon, it presents the Web Categories provided by the Reputation Service
      
      
   4. By selecting on a Suspicious Proxy record, you will highlight current row as well as the node from Network View frame. In addition, Details frame will be populated with additional communications directed to the same Proxy record.
      
      

3. The Network View frame
   Located at the top right corner, Network View is a hierarchical force graph used to represent the "Suspicious Proxy" connections.

   Network View Force Graph Order Hierarchy

   • Root Proxy Node
   • Proxy Request Method
   • Proxy Host
   • Proxy Path
   • Client IP Address
   
   

   Network View Functionality

   1. As soon as you move your mouse over a node, a dialog shows up providing additional information.

      

   2. Graph can be zoomed in/out and can be moved in the frame

      

   3. By double-clicking in the Root Proxy Node the graph can be fully expanded/collapsed

      

   4. By double-clicking a node, the node can be expanded/collapsed one level

      

   5. The path in yellow represents the Suspicious record selected in the suspicious frame

      

   6. Records will be highlighted with different colors depending upon the Risk Reputation provided by the Reputation Service

      

   7. A secondary mouse click over the Proxy Path or Client IP address nodes populates the Filter Box which eventually filter Suspicious & Network View Frames

      

4. The Details frame

   Located at the bottom of the Web page. It provides additional information for the selected connection in the Suspicious frame. It includes columns that are not part of the Suspicious frame such as User Agent, MIME Type, Proxy Server IP, Bytes.

   

5. The Scoring frame
   This frame contains a section where the Analyst can score Proxy records with different values. In order to assign a risk to a specific connection, select the correct rating (1=High risk, 2 = Medium/Potential risk, 3 = Low/Accepted risk) and click Score button.

   

The Score button

Pressing the 'Score' button will find all exact matches of the selected threat (Proxy Record) in the proxy_scores table and update their score to the value selected in the radio button list.

The Save button

Analysts must use the Save button in order to store the scored records into the proxy_threat_investigation table. After you click it, the rest of the frames in the page will be refreshed and the connections that you already scored will disappear on the suspicious connects page. At the same time, the scored connections will be made available for the ML to use as feedback. The following value will be obtained from the spot.conf file:

• LUSER

Proxy Threat Investigation

Walk-through

Access the analyst view for Proxy Suspicious Connects. Select the date that you want to review.

Your view should now look like this:

The analyst must previously score the suspicious connections before moving into Threat Investigation View, please refer to Proxy Suspicious Connects Analyst View walk-through.

Select Proxy > Threat Investigation from Apache Spot (incubating) Menu.

Threat Investigation Web Page will be opened, loading the embedded IPython notebook. A list with all Proxy Records scored as High risk will be presented

Expanded Search

Select any value from the list and press the "Search" button. The system will execute a query to the proxy table, looking into the raw data initially collected to find additional activity for the selected Proxy Record. Results will be extracted and displayed as a table in the UI. The quantity of results displayed on screen can be set by modifying the top_results variable, additional information on how to modify this variable can be found here

Save comments.

In addition, a web form is displayed under the title of 'Threat summary', where the analyst can enter a Title & Description on the kind of attack/behavior described by the particular Proxy Record that is under investigation.

Click on the Save button after entering the data to write it into the proxy_storyboard table.

Continue to the Storyboard.

Once you have saved comments on any suspicious IP or domain, you can continue to the Storyboard to check the results.

Proxy Storyboard

1. Select the option Proxy > Storyboard from Apache Spot (incubating) Menu.
   
   

2. Your view should look something like this, depending on how many threats you have analyzed and commented on the Threat Investigation page for that day. You can select a different date from the calendar.

   

Executive Threat Briefing

Executive Threat Briefing frame lists all the incident titles you entered at the Threat Investigation notebook. You can click on any title and view the comments at the bottom area of the panel.

Incident progression

Data source file: incident-progression-{id}.json
Incident progression frame is located on the right side of the Web page. Incident Progression displays a tree graph (dendrogram) detailing the type of connections that conform the activity related to the threat. It presents the following fields:

• Referer - URLs that refers to the Suspicious Proxy Record
• IP - All ip addresses connecting to the Suspicious Proxy Record
• Method - Proxy methods used to communicate in between the IP addresses and the Proxy Record
• ContentType - HTTP MIME Types
• Threat - Represents the Suspicious Proxy Record
• Referred - URLs that the Suspicious Proxy Record referred to

If multiple IP Addresses connects to a particular Proxy Threat (URL) you can scroll down/up, arrows indicate that there are more elements in the list.

Timeline

Timeline is created using the connections found during the Threat Investigation process. It will display 'clusters' of IP connections to the Proxy Record (URL), grouped by time; showing an overall idea of the times during the day with the most activity. You can zoom in or out into the graphs timeline using your mouse scroll. The number next to the IP Address represents the quantity of connections made from that particular IP to the Proxy Record in the displayed time.