Knowing is half the battle.

For new and old users who may benefit from the intricacies of Splunk. If you have tried to evaluate data where Datamodels do not exist or they do, you may have tried to coalesce data and failed.

The reason is because your values must exist between indexes e.g., index=fwall and index=awakesecurity. If host= rfc1918 in fwall and host=machine in awakesecurity and these values are unique to the two indexes you'll run into this problem every time.

For those of you veterans thinking, well ok Izzy I can just run transaction on my data sets and append with an eventstats command, yes I agree. But you better have one hell of a Splunk instance and your Ops guys better be primo. Transaction will tip the server over or keep you till next Christmas.

The reason I say this is because recently I encountered an organization where I couldn't get a lookup table to load to save anyone's life. In situations like these you need to improvise, especially when a 20 second query takes 8 hours of your work day and sometimes the dreaded Job Manager pops up. (LITERALLY 8 HOUR SEARCH)

I know you're thinking one of three things at this point, "inclusivity, field, or error delay label". For those who didn't read my previous post or know inclusivity is equal to everything left of the first pipe. Yes my searches do not use "*" why when you know your index, sourcetype, pattern, lookup etc., and ran tstats and metadata all before.

The point of this is knowing the battle. Use a shift left approach in a horrible situation like this. Export your two files to .csv when you've finally gone mad and have had enough, coalesce the data there and move along. This was my lesson learned from this experience, and no, field extraction is completely separate.

REACH OUT IF YOU WOULD LIKE A SPLUNK COLLABORATION.