Technology:
Splunk is the most
used server for collecting data from different sources, indexing the collected
data, analyzing the index data, preparing the reports based on indexed data.
The source of data may be from any environment and the source may be structured
format or unstructured format. Splunk provides some intelligent algorithms to
understand the indexed data used in security, IoT machine logs, and application
logs and provides the analytics of the data provided.
Splunk Data Sources: Splunk supplies a huge range of plugins to support different resources like for understanding windows event logs it supplies one plugin, for understanding Ubuntu machines logs it provides a various plugin. On the research of logs based Splunk will create the no of times the Operating system is loaded, the average CPU load in period, how many various users logged into the system, it will create the reports and it can also create a controlpanel for live data streaming.
Scalability: Splunk will not depend upon any outside services and it doesn`t necessity any database support, because Splunk internally uses a file system to store the indexed results. If the incoming data is increasing we can add another instance of the Splunk server to index and store the data. Splunk will internally communicate and it will distribute the incoming data and during the searching, it will collect the data from all Splunk servers and display it to the user. If we enable decrease then incoming data is distributed to more than one Splunk server.
Infinite control without losing granularity. Some monitoring products only permit you to keep so many months, weeks, or even days’ worth of data. Anothers decrease the granularity of older events, compressing various data points into one because of volume limits. The same is not true for Splunk. It can actually index hundreds of terabytes per day and hold practically limitless amounts of data.
Customize and extend Splunk with preferred technologies: Immediately change dashboards into HTML5 and create custom user experiences in HTML5 and JavaScript with the Splunk Web Framework. Use Python, Java, Ruby, C#, and PHP SDKs to combine Splunk capacity into developer workflows. Customize and extend Splunk with preferred technologies: Instantly convert dashboards into HTML5 and create custom user experiences in HTML5 and JavaScript with the Splunk Web Framework. Utilize Python, Java, JavaScript, Ruby, C#, and PHP SDKs to combine Splunk capacity into developer workflows.
Developer Tools: Splunk supplies plugins for Visual Studio for .NET-based applications, Eclipse and IntelliJ for Java-based applications, Splunk-logging plugin for Javascript, and it supplies plugin of Topmost of NodeJS, and also we can Splunk-Bunyan-logger plugin for javascript to gather HTTP Events.
The Splunk SDK for Java
ships with built-in support for IntelliJ for simple combination. Spring Source organized
the Spring Combination Extension for Splunk, making it simple for developers
using the famous Java framework Spring to log directly to Splunk from their
applications.
HTTP Event Collector (HEC): Splunk Event Collector is the fastest way to send data to Splunk Enterprises server and Splunk Cloud. HTTP Event Collector enables to send data over HTTP and HTTPS protocol directly to Splunk Enterprise and Splunk Cloud from application code. HEC provides the one rest endpoint to accept incoming data, in our application we can write to send data to the rest endpoint. HEC is authorized using tokens so that we don’t need to hard code the username and password details.
HTTP Event Gatherer
supplies a new way for developers to send application logging and metrics
directly to Splunk Enterprise and Splunk Cloud via HTTP in a highly important
and secure manner.
1. We can turn on HTTP Event Collector by enabling its endpoint, by default HEC is not enabled.
2.
Generate HEC Token.
3.
On the client end create POST request to
include authentication header or key/value pair to include HEC token.
4. POST data to HEC token receiver.
For Splunk Cloud instance it is not enabled, we can raise a request to Splunk cloud to enable HEC token.
Connecting to Splunk Server using Splunk SDK:
We can add the Splunk jar (http://download.splunk.com/misc/sdk/java/splunk-sdk-java-1.6.2.jar ) to project class path or we can download splunk-sdk from http://download.splunk.com/misc/sdk/java/splunk-sdk-java-1.6.2.zip and we can extract splunk jar from splunk-sdk-java/dist folder.
Below the sample java program to connect to Splunk server and
printing the application on it.
importcom.splunk.*;
public class SplunkTest {
public static void main(String[] args) {
// Create a map of arguments and add login parameters
ServiceArgsloginArgs = new ServiceArgs();
loginArgs.setUsername(“admin”);
loginArgs.setPassword(“changeme”);
loginArgs.setHost(“localhost”);
loginArgs.setPort(8089);
// Create a Service instance and log in with the argument map
Service service = Service.connect(loginArgs);
// Print installed apps to the console to verify login
for (Application app : service.getApplications().values()) {
System.out.println(app.getName());
}
}
}
We can either user ServiceArgs or Service constructor to pass host
and port details.
importcom.splunk.*;
public class SplunkTest {
public static void main(String[] args) {
// A second way to create a new Service object and log in
Service service = new Service(“localhost”, 8089);
service.login(“admin”, “changeme”);
// Print installed apps to the console to verify login
for (Application app : service.getApplications().values()) {
System.out.println(app.getName());
}
}
}
Using same token across all applications:
Service class creates token after login, it will use this token
for all subsequent requests.
importcom.splunk.*;
import com.sun.org.apache.xerces.internal.impl.dv.util.Base64;
public class SplunkTest {
public static void main(String[] args) {
Service service = new Service(“localhost”, 8089);
String credentials = “admin:changeme”;
String basicAuthHeader = Base64.encode(credentials.getBytes());
service.setToken(“Basic ” + basicAuthHeader);
// Print the session token
System.out.println(“Your session token: ” + service.getToken());
// Print installed apps to the console to verify login
for (Application app : service.getApplications().values()) {
System.out.println(app.getName());
}
}
}
For example, if you wish to use the same session token on various
customers that are retrieving a single Splunk Enterprise instance, you can log
in on one customer, get the session token applying the Service.getToken method,
and after pass that token to another customers to use when they log in (use the
Service.setToken method).We can also merge to Splunk Enterprise by accessing primary
access approve, for example if you have various Splunk Enterprise instances
behind a load balancer and wish to use the same session token for all. To build
a token using a primary access header, Base64-encode a string with your authorization
in the format “username: password” and prepend the authorization method
(“Basic”).
Integrating
Splunk with Spring Applications:
The Spring Integration adapter for Splunk includes 2 adapters:
·
Inbound Channel Adapter to find data from Splunk.
·
Outbound Channel Adapter to force event data into Splunk.
Spring integration gives namespace for splunk, it will acesses the
spring integration adapter for splunk internally.
Generating beans for splunk server object.
<?xml version=”1.0″ encoding=”UTF-8″?>
<beans xmlns=”http://www.springframework.org/schema/beans”
xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
xmlns:int-splunk=”http://www.springframework.org/schema/integration/splunk”
xsi:schemaLocation=”http://www.springframework.org/schema/integration/splunk
http://www.springframework.org/schema/integration/splunk/spring-integration-splunk.xsd
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd”>
<int-splunk:server id=”splunkServer” host=”somehost”
port=”8089″ userName=”user” password=”password” owner=”admin”/>
</beans>
Outbound Channel Adapter: Outbound channel adapter is used to put
data into Splunk from channels in Spring Integration. There are 3 kinds of
method to put data:
Submit (HTTP REST)
Stream
Tcp
The major dissimilarity using the REST inputs vs plain TCP/UDP
inputs is truly in the Splunk event handling pipeline.
With REST, you have to announce your event metadata (index,
source, source type…) in the HTTP request at the source. You can’t truly change
the log event more after you have made and sent it to Splunk. Typically though,
for customers using REST, this is fine because they are well shaping their log
events before sending them anyway: no more processing and manipulation is needed.
<int-splunk:outbound-channel-adapter
id=”splunkOutboundChannelAdapter” auto-startup=”true” order=”1″
channel=”outputToSplunk” splunk-server-ref=”splunkServer”
pool-server-connection=”true” sourceType=”spring-integration” source=”example”
ingest=”submit”>
</int-splunk:outbound-channel-adapter>
With TCP inputs, modify the event data in Splunk before it gets
indexed (using entrys in props.conf/transforms.conf). The event metadata
(index, source, source type…) gets announced on the Splunk side when you start
the TCP/UDP input and can also be dynamically created,so essentially you have a
lot more control over the indexing of the event data.
<int-splunk:outbound-channel-adapter
id=”splunkOutboundChannelAdapter” auto-startup=”true” order=”1″
channel=”outputToSplunk” splunk-server-ref=”splunkServer” ingest=”tcp”
tcpPort=”9999″/>
To use outbound channel adapter with stream, you can define the
adapter as following:
<int-splunk:outbound-channel-adapter
id=”splunkOutboundChannelAdapter” auto-startup=”true” order=”1″ channel=”outputToSplunk”
splunk-server-ref=”splunkServer” ingest=”stream”/>
Inbound
channel adapter is used to get data out of Splunk and put into Spring
Integration’s channel. There are 5 ways to get data out of Splunk:
·
Blocking
·
Non blocking
·
Saved search
·
Realtime
·
Export
<int-splunk:inbound-channel-adapter
id=”splunkInboundChannelAdapter” auto-startup=”true” search=”search
spring:example” splunk-server-ref=”splunkServer” channel=”inputFromSplunk”
mode=”blocking” initEarliestTime=”-1d”>
<int:poller fixed-rate=”5″ time-unit=”SECONDS”/>
</int-splunk:inbound-channel-adapter>
Where mode is one of the above 5 types.
Using log frameworks to log into splunk servers.
Splunklogging provides the jar provides appenders for most used
logger frameworks.
java.util.logging handler for logging to Splunk REST endpoints
java.util.logging handler for logging to Splunk Raw TCP Server
Socket
java.util.logging handler for logging to Splunk HEC Endpoint
Log4j appender for logging to Splunk REST endpoints
Log4j appender for logging to Splunk Raw TCP Server Socket
Log4j appender for logging to Splunk HEC Endpoint
Logbackappender for logging to Splunk REST endpoints
Logbackappender for logging to Splunk Raw TCP Server Socket
Logbackappender for logging to Splunk HEC Endpoint
Log4j 2 appender for logging to Splunk HEC Endpoint
Log4j 2 config examples for TCP and UDP logging
For example if we want to use Rest services we can include below
configuration in logger.properties.
log4j.appender.splunkrest=com.splunk.logging.log4j.appender.SplunkRestAppender
log4j.appender.splunkrest.user=admin
log4j.appender.splunkrest.pass=somepass
log4j.appender.splunkrest.host=localhost
log4j.appender.splunkrest.port=8089
log4j.appender.splunkrest.delivery=stream
log4j.appender.splunkrest.metaSource=rest
log4j.appender.splunkrest.metaSourcetype=testing
log4j.appender.splunkrest.metaIndex=main
log4j.appender.splunkrest.maxQueueSize=5MB
log4j.appender.splunkrest.dropEventsOnQueueFull=false
So that it will be logged automatically.
Conclusion:
In this article, java developmentcompany India have clarified the uses of Splunk in Java. Splunk is the
software to receive the log messages and do index using log messages and supplies
the metrics using log messages. It gives the SDK and logging frameworks for
connecting to Splunk Enterprises and Splunk Cloud. Spring Integration gives an
adapter for connecting Splunk severs also it gives a namespace for generating Splunk-related
beans.
Related Aricle:
How to Wrap Text inside the Column in SWT Java-Based Framework?
0 comments:
Post a Comment