This is a good option that is feasible and also fulfils the purpose. But the complexity rises when there are issues around network connectivity, cross data centre chatter and security concerns from data egress. This makes it difficult for this kind of solution to be put in place.
Moreover, there can be limitations of the Ip Address, CIDR and hostname conventions followed across different sites.
Since both the usual suspects have been ruled out to tackle this problem, it is time to think out of the box. This option uses Splunk’s built in “indexAndForward” capability to deal with the problem. Also it makes sure that each cluster on it’s own has access to it’s own internal data AND has a copy of data sent over to the Central Monitoring Console (Yes, what an orgininal name! )
Following are the default parameters for whitelist and blacklist in the
With the above configuration, Splunk forwards all the data (both Splunk logs and data sources coming in)
To address the problem, we have to whitelist only internal indexes (_*) on the indexers outputs.conf. Post the below change, Splunk will only forward the data coming in _internal,_audit,_introspection, _telemetry.
On the Indexer, make the following change to SPLUNK_HOME/etc/system/local/outputs.conf.