Splunk Interview Questions and Answers

Splunk is a powerful platform for searching, analyzing, monitoring, visualizing and reporting of your enterprise data. It acquires important machine data and then converts it into powerful operational intelligence by giving real time insight to your data using alerts, dashboards and charts etc.

Splunk works into three phases-

• In the first phase, it gathers data to solve your query from many sources as required.
• Secondly, it converts that data into results that can solve your query.
• Third, it displays the information/answers via a chart, report or graph, which is understood by large audiences.

Splunk has four important components:

Indexer – It indexes the machine data

Forwarder – Refers to Splunk instances that forward data to the remote indexers

Search Head – Provides GUI for searching

Deployment Server – Manages the Splunk components like indexer, forwarder, and search head in computing environment

Splunk has two types of Splunk forwarder which are as follows:

Universal Forwarders – It performs processing on the incoming data before forwarding it to the indexer.

Heavy Forwarders – It parses the data before forwarding them to the indexer works as an intermediate forwarder, remote collector.

An alert is an action that a saved search triggers on regular intervals set over a time range, based on the results of the search. When the alerts are triggered, various actions occur consequently. For instance, sending an email when a search to the predefined list of people is triggered.

Three types of alerts:

Pre-result alerts: Most commonly used alert type and runs in real-time for an all- time span. These alerts are designed such that whenever a search returns a result, they are triggered.

Scheduled alerts: The second most common- scheduled results are set up to evaluate the results of a historical search result running over a set time range on a regular schedule. You can define a time range, schedule and the trigger condition to an alert.

Rolling-window alerts: These are the hybrid of pre-result and scheduled alerts. Similar to the former, these are based on real-time search but do not trigger each time the search returns a matching result. It examines all events in real-time happing within the rolling window and triggers the time that specific condition by that event in the window is met, like the scheduled alert is triggered on a scheduled search.

SPL commands are divided into five categories:

1. Sorting Results – Ordering results and (optionally) limiting the number of results.
2. Filtering Results – It takes a set of events or results and filters them into a smaller set of results.
3. Grouping Results – Grouping events so you can see patterns.
4. Filtering, Modifying and Adding Fields – Taking search results and generating a summary for reporting.
5. Reporting Results – Filtering out some fields to focus on the ones you need, or modifying or adding fields to enrich your results or events.

Common ports numbers on which services are run (by default) are:

Service Port Number

• Splunk Management Port: 8089
• Splunk Index Replication Port 8080
• KV store 8191
• Splunk Web Port: 8000
• Splunk Indexing Port: 9997
• Splunk network port: 514

A directory that contains indexed data is known as a Splunk bucket. It also contains events of a certain period. Bucket lifecycle includes following stages:

• Hot – It contains newly indexed data and is open for writing. For each index, there are one or more hot buckets available
• Warm – Data rolled from hot
• Cold – Data rolled from warm
• Frozen – Data rolled from cold. The indexer deletes frozen data by default but users can also archive it.
• Thawed – Data restored from an archive. If you archive frozen data, you can later return it to the index by thawing (defrosting) it.

To enable Splunk to boot start use the following command:

$SPLUNK_HOME/bin/splunk enable boot-start

To disable Splunk to boot start use the following command:

$SPLUNK_HOME/bin/splunk disable boot-start

It evaluates an expression and consigns the resulting value into a destination field. If the destination field matches with an already existing field name, the existing field is overwritten with the eval expression. This command evaluates Boolean, mathematical and string expressions.

Using eval command:

• Convert Values
• Round Values
• Perform Calculations
• User conditional statements
• Format Values

The lookup command adds fields based while looking at the value in an event, referencing a lookup table, and adding the fields in matching rows in the lookup table to your event.

Example:

… | lookup usertogroup user as local_user OUTPUT group as user_group

inputlookup command returns the whole lookup table as search results.

For example

…| inputlookup intellipaatlookup returns a search result for every row in the table intellipaatlookup which has two field values:

• host
• machine_type.

This command outputs the current search results to a lookup table on the disk.

For example

…| outputlookup intellipaattable.csv saves all the results into intellipaattable.csv.

• where – Evaluates an expression for filtering results. If the evaluation is successful and the result is TRUE, the result is retained; otherwise, the result is discarded.
• dedup – Removes subsequent results that match specified criteria.
• head – Returns the first count results. Using head permits a search to stop retrieving events from disk when it finds the desired number of results.
• tail – Unlike head command, this returns the last results

• top – Finds most frequent tuple of values of all fields in the field list along with the count and percentage.
• rare – Finds least frequent tuple of values of all fields in the field list.
• stats – Calculates aggregate statistics over a dataset
• chart – Creates tabular data output suitable for charting
• timechart – Creates a time series chart with corresponding table of statistics.

transaction – Groups events that meet different constraints into transactions, where transactions are the collections of events possibly from multiple sources.

It sorts search results by the specified fields.

Syntax:

sort [<count>] <sort-by-clause>… [desc]

Example:

… | sort num(ip), -str(url)

It sort results by ip value in ascending order whereas url value in descending order.

Search head pooling is a group of connected servers that are used to share load, Configuration and user data Whereas Search head clustering is a group of Splunk Enterprise search heads used to serve as a central resource for searching. Since the search head cluster supports member interchangeability, the same searches and dashboards can be run and viewed from any member of the cluster.

Alert manager displays the list of most recently fired alerts, i.e. alert instances. It provides a link to view the search results from that triggered alert. It also displays the alert’s name, app, type (scheduled, real-time, or rolling window), severity and mode.

SOS stands for Splunk on Splunk. It is a Splunk app that provides graphical view of your Splunk environment performance and issues.

It has following purposes:

• Diagnostic tool to analyze and troubleshoot problems
• Examine Splunk environment performance
• Solve indexing performance issues
• Observe scheduler activities and issues
• See the details of scheduler and user driven search activity
• Search, view and compare configuration files of Splunk

It is a general SQL database plugin that permits you to easily combine database information with Splunk queries and reports. It provides reliable, scalable and real-time integration between Splunk Enterprise and relational databases.

Splunk App Framework resides within Splunk’s web server and permits you to customize the Splunk Web UI that comes with the product and develop Splunk apps using the Splunk web server. It is an important part of the features and functionalities of Splunk Software, which does not license users to modify anything in the Splunk Software.

Splunk SDKs are designed to allow you to develop applications from the ground up and not require Splunk Web or any components from the Splunk App Framework. These are separately licensed to you from the Splunk Software and do not alter the Splunk Software.

The indexer is a Splunk Enterprise component that creates and manages indexes. The main functions of an indexer are:

• Indexing incoming data
• Searching indexed data

Splunk indexer has following stages:

Input: Splunk Enterprise acquires the raw data from various input sources and breaks it into 64K blocks and assign them some metadata keys. These keys include host, source and source type of the data.

Parsing: Also known as event processing, during this stage, the Enterprise analyzes and transforms the data, breaks data into streams, identifies, parses and sets timestamps, performs metadata annotation and transformation of data.

Indexing: In this phase, the parsed events are written on the disk index including both compressed data and the associated index files.

Searching: The ‘Search’ function plays a major role during this phase as it handles all searching aspects (interactive, scheduled searches, reports, dashboards, alerts) on the indexed data and stores saved searches, events, field extractions and views

Replace command performs a search-and-replace on specified field values with replacement values. The values in a search and replace are case sensitive.

Syntax:

replace (<wc-string> WITH <wc-string>)… [IN <field-list>]

Example:

… | replace *localhost WITH localhost IN host

Change any host value that ends with “localhost” to “localhost”.

File precedence in Splunk is as follows:

• System local directory: top priority
• App local directories
• App default directories
• System default directory: lowest priority

It removes results that do not match the specified regular expression.

Syntax:

regex (<field>=<regex-expression> | <field>!=<regex-expression> | <regex-expression>)

Splunk default configuration is stored at  $splunkhome/etc/system/default

To reset password, follow these steps:

• Log in to server on which Splunk is installed
• Rename password file at $splunk-home\etc\passwd
• Restart Splunk
• After restart, you can login using default username: admin password: changeme

Using syntax: rest /servicesNS/-/-/saved/searches splunk_server=loca

stats – This command produces summary statistics of all existing fields in your search results and store them as values in new fields.

eventstats – It is same as stats command except that aggregation results are added in order to every event and only if the aggregation is applicable to that event. It computes the requested statistics similar to stats but aggregates them to the original raw data.