Home
Load Balancing
Sending F5 Logs via High Speed Logging to Splunk

Sending F5 Logs via High Speed Logging to Splunk

Kerry Cordero
Load BalancingSecurity
4 min read

What is HSL?

F5 High-Speed Logging (HSL) is a mechanism that F5 devices, like BIG-IP, use to log and send detailed information about transactions at a high rate to a remote syslog server or an analytics system like Splunk. HSL is designed to handle a high volume of logs while minimizing the performance impact on the BIG-IP system.

Configuration via the GUI

(a) Create a pool of remote log servers to which the BIG-IP system can send log messages

  1. Log in to the Configuration utility.
  2. Navigate to Local Traffic > Pools
  3. Click Create.
  4. In the Name field, type a unique name for the pool.
  5. Using the New Members setting, add the IP address for each remote logging server that you want to include in the pool:
  6. Type an IP address in the Address field, or select a node address from the Node List.
  7. Type a service number in the Service Port field, or select a service name from the list.
    Note: Typical remote logging servers require port 514.
  8.  Click Add
  9. Click Finished.

(b) Create a remote high-speed log destination

  1. Navigate to System > Logs > Configuration > Log Destinations .
  2. Click Create.
  3. In the Name field, type a unique, identifiable name for this destination.
  4. From the Type list, select Remote High-Speed Log.
    Note: Since we will be sending the logs to Splunk which require data be sent to the Splunk server in a specific format, you must create an additional log destination of the required type, and associate it with a log destination of the Remote High-Speed Log type. With this configuration, the BIG-IP system can send data to the servers in the required format.
  5. From the Pool Name list, select the pool of remote log servers to which you want the BIG-IP system to send log messages.
  6. From the Protocol list, select the protocol used by the high-speed logging pool members.
  7. Click Finished.
     Note: With this configuration, BIG-IP system is configured to send an unformatted string of text to the log servers.

(c) Create a formatted remote high-speed log destination for Splunk

  1. Navigate to System > Logs > Configuration > Log Destinations .
  2. Click Create.
  3. In the Name field, type a unique, identifiable name for this destination.
  4. From the Type list, select Splunk.
  5. From the Forward To list, select remote high-speed log destination to which you want the BIG-IP system to send log messages.
  6. Click Finished.

(d) Create a Publisher

  1. Navigate to System > Logs > Configuration > Log Publishers .
  2. Click Create.
  3. In the Name field, type a unique, identifiable name for this publisher.
  4. For the Destinations setting, select the Splunk destination from the Available list, and click << to move the destination to the Selected list..
  5. Click Finished.

(e) Creating a logging filter

  1. Navigate to System > Logs > Configuration > Log Filters
  2. In the Name field, type a unique, identifiable name for this filter.
  3. From the Severity list, select the level of alerts that you want the system to use for this filter.
  4. From the Source list, select the system processes from which messages will be sent to the log.
  5. In the Message ID field, type the first eight hex-digits of the specific message ID that you want the system to include in the log. Use this field when you want a log to contain only each instance of one specific log message.
  6. From the Log Publisher list, select the publisher that includes the destinations to which you want to send log messages.
  7. Click Finished

Step-by-step Breakdown

a. Creating a Pool of Remote Log Servers

In this step, you’re creating a pool of remote servers (like Splunk) to which BIG-IP can send log messages. A pool groups together one or more log servers, so logs can be sent to multiple destinations simultaneously to ensure redundancy and load balancing.

b. Creating a Remote High-Speed Log Destination

Here, you’re creating a destination configuration specifying where (the pool created in step a) and how (protocol type) the log messages should be sent. At this stage, the logs are being sent as unformatted text strings.

c. Creating a Formatted Remote High-Speed Log Destination for Splunk

This step adds formatting to the log messages to meet Splunk’s requirements. Essentially, it makes sure the logs BIG-IP sends are structured in a way that Splunk can understand and process efficiently.

d. Creating a Publisher

In this stage, a publisher is created to define how the logs should be sent to Splunk. The publisher uses the formatted log destination created in step c to send logs.

e. Creating a Logging Filter

Finally, you create a filter to specify which messages get logged based on severity and source, and direct them to the appropriate publisher. This helps in reducing noise in the logs by filtering out unnecessary messages and only logging messages that meet certain criteria.

Recommendations and Best Practices

  1. Naming Conventions: Use clear and descriptive names for pools, destinations, publishers, and filters to easily identify their purpose later on.
  2. Log Server Pool: When setting up the pool of log servers, it’s good to add redundancy by having more than one server in the pool to ensure high availability.
  3. Protocol Selection: While selecting the protocol in step b, it is common to use UDP for high-speed logging because of its lower overhead compared to TCP. However, keep in mind that UDP is less reliable than TCP.
  4. Splunk Configuration: Before proceeding with these steps, ensure that your Splunk server is correctly configured to receive data over the designated port and in the format specified in step c.
  5. Filter Configuration: When setting up filters in step e, it would be best to fine-tune the severity and source settings over time based on the kind of log messages you find most useful.
  6. Testing: After the configuration, test the setup extensively to ensure that it meets your requirements and that Splunk is receiving and correctly interpreting the log messages.
  7. Documentation: Document the setup process and any specific configurations, so future administrators can understand the setup and make informed decisions about any changes.
  8. Maintenance: Regularly review and maintain the logging setup to ensure it continues to meet your needs, updating the filters as necessary to adapt to changes in your environment.

By following the steps and considerations above, you’ll set up a robust logging solution leveraging F5 BIG-IP’s high-speed logging mechanism to send detailed log data to Splunk, facilitating advanced analytics and monitoring.

Source:
https://my.f5.com/manage/s/article/K15316506

© 2024 cordero.me • All Rights Reserved