Technology:
Splunk is the most
used server for collecting data from different sources, indexing the collected
data, analyzing the index data, preparing the reports based on indexed data.
The source of data may be from any environment and the source may be structured
format or unstructured format. Splunk provides some intelligent algorithms to
understand the indexed data used in security, IoT machine logs, and application
logs and provides the analytics of the data provided.
Splunk
Data Sources: Splunk supplies a huge range of plugins to support different
resources like for understanding windows event logs it supplies one plugin, for
understanding Ubuntu machines logs it provides a various plugin. On the research
of logs based Splunk will create the no of times the Operating system is
loaded, the average CPU load in period, how many various users logged into the
system, it will create the reports and it can also create a controlpanel for
live data streaming.
Scalability: Splunk
will not depend upon any outside services and it doesn`t necessity any database
support, because Splunk internally uses a file system to store the indexed
results. If the incoming data is increasing we can add another instance of the
Splunk server to index and store the data. Splunk will internally communicate
and it will distribute the incoming data and during the searching, it will
collect the data from all Splunk servers and display it to the user. If we
enable decrease then incoming data is distributed to more than one Splunk
server.
Infinite control without
losing granularity. Some monitoring products only permit you to keep so many
months, weeks, or even days’ worth of data. Anothers decrease the granularity
of older events, compressing various data points into one because of volume
limits. The same is not true for Splunk. It can actually index hundreds of
terabytes per day and hold practically limitless amounts of data.
Customize
and extend Splunk with preferred technologies: Immediately change
dashboards into HTML5 and create custom user experiences in HTML5 and
JavaScript with the Splunk Web Framework. Use Python, Java, Ruby, C#, and PHP
SDKs to combine Splunk capacity into developer workflows. Customize and extend
Splunk with preferred technologies: Instantly convert dashboards into HTML5 and
create custom user experiences in HTML5 and JavaScript with the Splunk Web
Framework. Utilize Python, Java, JavaScript, Ruby, C#, and PHP SDKs to combine
Splunk capacity into developer workflows.
Developer
Tools: Splunk supplies plugins for Visual Studio for .NET-based
applications, Eclipse and IntelliJ for Java-based applications, Splunk-logging
plugin for Javascript, and it supplies plugin of Topmost of NodeJS, and also we
can Splunk-Bunyan-logger plugin for javascript to gather HTTP Events.
The Splunk SDK for Java
ships with built-in support for IntelliJ for simple combination. Spring Source organized
the Spring Combination Extension for Splunk, making it simple for developers
using the famous Java framework Spring to log directly to Splunk from their
applications.
HTTP
Event Collector (HEC): Splunk Event Collector is the fastest way to
send data to Splunk Enterprises server and Splunk Cloud. HTTP Event Collector
enables to send data over HTTP and HTTPS protocol directly to Splunk Enterprise
and Splunk Cloud from application code. HEC provides the one rest endpoint to
accept incoming data, in our application we can write to send data to the rest
endpoint. HEC is authorized using tokens so that we don’t need to hard code the
username and password details.
HTTP Event Gatherer
supplies a new way for developers to send application logging and metrics
directly to Splunk Enterprise and Splunk Cloud via HTTP in a highly important
and secure manner.
Enabling
HTTP Event Collector:
1.
We can turn on HTTP Event Collector by
enabling its endpoint, by default HEC is not enabled.
2.
Generate HEC Token.
3.
On the client end create POST request to
include authentication header or key/value pair to include HEC token.
4.
POST data to HEC token receiver.
For Splunk Cloud instance
it is not enabled, we can raise a request to Splunk cloud to enable HEC token.
Connecting
to Splunk Server using Splunk SDK:
We can add the Splunk jar (http://download.splunk.com/misc/sdk/java/splunk-sdk-java-1.6.2.jar )
to project class path or we can download splunk-sdk from http://download.splunk.com/misc/sdk/java/splunk-sdk-java-1.6.2.zip and
we can extract splunk jar from splunk-sdk-java/dist folder.
Below the sample java program to connect to Splunk server and
printing the application on it.
importcom.splunk.*;
public class SplunkTest {
public static void main(String[] args) {
// Create a map of arguments and add login parameters
ServiceArgsloginArgs = new ServiceArgs();
loginArgs.setUsername(“admin”);
loginArgs.setPassword(“changeme”);
loginArgs.setHost(“localhost”);
loginArgs.setPort(8089);
// Create a Service instance and log in with the argument map
Service service = Service.connect(loginArgs);
// Print installed apps to the console to verify login
for (Application app : service.getApplications().values()) {
System.out.println(app.getName());
}
}
}
We can either user ServiceArgs or Service constructor to pass host
and port details.
importcom.splunk.*;
public class SplunkTest {
public static void main(String[] args) {
// A second way to create a new Service object and log in
Service service = new Service(“localhost”, 8089);
service.login(“admin”, “changeme”);
// Print installed apps to the console to verify login
for (Application app : service.getApplications().values()) {
System.out.println(app.getName());
}
}
}
Using same token across all applications:
Service class creates token after login, it will use this token
for all subsequent requests.
importcom.splunk.*;
import com.sun.org.apache.xerces.internal.impl.dv.util.Base64;
public class SplunkTest {
public static void main(String[] args) {
Service service = new Service(“localhost”, 8089);
String credentials = “admin:changeme”;
String basicAuthHeader = Base64.encode(credentials.getBytes());
service.setToken(“Basic ” + basicAuthHeader);
// Print the session token
System.out.println(“Your session token: ” + service.getToken());
// Print installed apps to the console to verify login
for (Application app : service.getApplications().values()) {
System.out.println(app.getName());
}
}
}
For example, if you wish to use the same session token on various
customers that are retrieving a single Splunk Enterprise instance, you can log
in on one customer, get the session token applying the Service.getToken method,
and after pass that token to another customers to use when they log in (use the
Service.setToken method).We can also merge to Splunk Enterprise by accessing primary
access approve, for example if you have various Splunk Enterprise instances
behind a load balancer and wish to use the same session token for all. To build
a token using a primary access header, Base64-encode a string with your authorization
in the format “username: password” and prepend the authorization method
(“Basic”).
Integrating
Splunk with Spring Applications:
The Spring Integration adapter for Splunk includes 2 adapters:
·
Inbound Channel Adapter to find data from Splunk.
·
Outbound Channel Adapter to force event data into Splunk.
Spring integration gives namespace for splunk, it will acesses the
spring integration adapter for splunk internally.
Generating beans for splunk server object.
<?xml version=”1.0″ encoding=”UTF-8″?>
<beans xmlns=”http://www.springframework.org/schema/beans”
xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xmlns:int-splunk=”http://www.springframework.org/schema/integration/splunk”
xsi:schemaLocation=”http://www.springframework.org/schema/integration/splunk
http://www.springframework.org/schema/integration/splunk/spring-integration-splunk.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd”>
<int-splunk:server id=”splunkServer” host=”somehost”
port=”8089″ userName=”user” password=”password” owner=”admin”/>
</beans>
Outbound Channel Adapter: Outbound channel adapter is used to put
data into Splunk from channels in Spring Integration. There are 3 kinds of
method to put data:
Submit (HTTP REST)
Stream
Tcp
The major dissimilarity using the REST inputs vs plain TCP/UDP
inputs is truly in the Splunk event handling pipeline.
With REST, you have to announce your event metadata (index,
source, source type…) in the HTTP request at the source. You can’t truly change
the log event more after you have made and sent it to Splunk. Typically though,
for customers using REST, this is fine because they are well shaping their log
events before sending them anyway: no more processing and manipulation is needed.
<int-splunk:outbound-channel-adapter
id=”splunkOutboundChannelAdapter” auto-startup=”true” order=”1″
channel=”outputToSplunk” splunk-server-ref=”splunkServer”
pool-server-connection=”true” sourceType=”spring-integration” source=”example”
ingest=”submit”>
</int-splunk:outbound-channel-adapter>
With TCP inputs, modify the event data in Splunk before it gets
indexed (using entrys in props.conf/transforms.conf). The event metadata
(index, source, source type…) gets announced on the Splunk side when you start
the TCP/UDP input and can also be dynamically created,so essentially you have a
lot more control over the indexing of the event data.
<int-splunk:outbound-channel-adapter
id=”splunkOutboundChannelAdapter” auto-startup=”true” order=”1″
channel=”outputToSplunk” splunk-server-ref=”splunkServer” ingest=”tcp”
tcpPort=”9999″/>
To use outbound channel adapter with stream, you can define the
adapter as following:
<int-splunk:outbound-channel-adapter
id=”splunkOutboundChannelAdapter” auto-startup=”true” order=”1″ channel=”outputToSplunk”
splunk-server-ref=”splunkServer” ingest=”stream”/>
Inbound
channel adapter is used to get data out of Splunk and put into Spring
Integration’s channel. There are 5 ways to get data out of Splunk:
·
Blocking
·
Non blocking
·
Saved search
·
Realtime
·
Export
<int-splunk:inbound-channel-adapter
id=”splunkInboundChannelAdapter” auto-startup=”true” search=”search
spring:example” splunk-server-ref=”splunkServer” channel=”inputFromSplunk”
mode=”blocking” initEarliestTime=”-1d”>
<int:poller fixed-rate=”5″ time-unit=”SECONDS”/>
</int-splunk:inbound-channel-adapter>
Where mode is one of the above 5 types.
Using log frameworks to log into splunk servers.
Splunklogging provides the jar provides appenders for most used
logger frameworks.
java.util.logging handler for logging to Splunk REST endpoints
java.util.logging handler for logging to Splunk Raw TCP Server
Socket
java.util.logging handler for logging to Splunk HEC Endpoint
Log4j appender for logging to Splunk REST endpoints
Log4j appender for logging to Splunk Raw TCP Server Socket
Log4j appender for logging to Splunk HEC Endpoint
Logbackappender for logging to Splunk REST endpoints
Logbackappender for logging to Splunk Raw TCP Server Socket
Logbackappender for logging to Splunk HEC Endpoint
Log4j 2 appender for logging to Splunk HEC Endpoint
Log4j 2 config examples for TCP and UDP logging
For example if we want to use Rest services we can include below
configuration in logger.properties.
log4j.appender.splunkrest=com.splunk.logging.log4j.appender.SplunkRestAppender
log4j.appender.splunkrest.user=admin
log4j.appender.splunkrest.pass=somepass
log4j.appender.splunkrest.host=localhost
log4j.appender.splunkrest.port=8089
log4j.appender.splunkrest.delivery=stream
log4j.appender.splunkrest.metaSource=rest
log4j.appender.splunkrest.metaSourcetype=testing
log4j.appender.splunkrest.metaIndex=main
log4j.appender.splunkrest.maxQueueSize=5MB
log4j.appender.splunkrest.dropEventsOnQueueFull=false
So that it will be logged automatically.
Conclusion:
In this article, java developmentcompany India have clarified the uses of Splunk in Java. Splunk is the
software to receive the log messages and do index using log messages and supplies
the metrics using log messages. It gives the SDK and logging frameworks for
connecting to Splunk Enterprises and Splunk Cloud. Spring Integration gives an
adapter for connecting Splunk severs also it gives a namespace for generating Splunk-related
beans.
Related Aricle:
How to Wrap Text inside the Column in SWT Java-Based Framework?
Java Determines Success
