One Schema for Every Source

Why it matters

An investigation rarely stays in one source. The same host can appear in a network record, a Windows event and a Linux audit record. If each source named that host differently, you would need a different search for each one.

The Elastic Common Schema, ECS, is Elastic's answer: a shared set of field names that every source is mapped onto. Knowing it tells you which field to search before you look, and knowing where a source departs from it saves you from empty results that mean nothing.

What analysts see

In Discover, ECS shows up in three ways.

• Field names grouped by what they describe: host., user., process., source., destination. and event..
• The same field name on records from different sources, so one query matches all of them.
• A source's own native field names alongside the ECS ones, where the mapping keeps them.

The field list for your data view tells you which of these you're dealing with. Read it before you search.

Core concept

ECS groups fields by what they describe. host.* fields describe the machine that recorded the event, user.* the account involved, process.* the program, source.* and destination.* the two ends of a connection, and event.* the record itself: what kind it is and how it turned out.

Every source is mapped onto those names when it is collected. A Windows workstation, a domain controller and a Linux server can all fill host.name, so one search on host.name finds a host's records whatever wrote them.

Mapping is not always complete. Some of our sources keep their original field names next to the ECS ones: on Windows, the event number appears both as event.code and as winlog.event_id. Other sources map everything and keep no native names. Neither is wrong; check which you have.

Not every record fills every field. A field can exist in the schema and still be empty on a given kind of record. An account name is missing on many machine-level events, and Linux records carry no event code at all; they use event.action to say what kind of record they are. A field that is often empty is not evidence of anything.

event.action and event.outcome describe the record itself: what happened, and whether it succeeded, failed or neither applies. How often they are filled varies by source, so read them where they exist and don't rely on them where they don't.

Authentication records follow the same pattern. Whatever the source, an authentication record describes an account, what it tried to reach, where the attempt came from, the method and the result. The host that writes the record is not always the host the person was using, so check which host a field describes before you connect two records.

Which hosts write which records matters too. On our ranges, domain controllers keep their own records, separate from every other Windows host's, and Linux hosts record their activity through an audit agent. If a record you expect isn't there, check whether that kind of host writes it at all.

Annotated examples

How to investigate

1. 1.

   Read the field list first.

   For each data view, note which ECS groups it fills, and whether native names appear alongside.

2. 2.

   Search the ECS name.

   It spans every source that maps onto it.

   host.name: "ws-120.corp.example"

3. 3.

   If an ECS field returns nothing, check the source.

   The field may be empty for that kind of record, or the value may sit only under a native name.

   winlog.computer_name: "ws-120.corp.example"

4. 4.

   On records without an event code, read event.action.

   It names the kind of record, as on Linux.

5. 5.

   Check which host wrote the record.

   Compare host.name with the host the activity is about before you connect two records.

6. 6.

   Treat a sparse field as missing information.

   An empty field is not a finding.

Common mistakes

• Mistake: Searching one source's native name and missing the others.

  InsteadSearch the ECS name, which every mapped source shares.

• Mistake: Reading an empty field as "nothing happened".

  InsteadMany fields are often empty for a kind of record. Check how often the field is filled before you read meaning into it.

• Mistake: Looking for an event code on Linux records.

  InsteadUse event.action, which names the kind of record.

• Mistake: Assuming the host in host.name is where the person was.

  InsteadIt is the host that wrote the record. Check which host each field describes.

• Mistake: Assuming every case maps its data the same way.

  InsteadRead the field list for each case's data views.

Field references

Check yourself

1. 1.
   You want every record about one host, from all sources. Which field do you search?

   • A.winlog.computer_name
   • B.host.name
   • C.source.ip
   • D.event.code

2. 2.
   A Linux record has no event.code. Where do you find what kind of record it is?

   • A.winlog.event_id
   • B.process.name
   • C.event.action
   • D.Nowhere: it has no type

3. 3.
   user.name is empty on many records from one source. What does that tell you?

   • A.That this source often records no account for that kind of record
   • B.That someone removed the account names
   • C.That no user was involved in anything
   • D.That the source is not working

Apply it in a Case

On your next case, open the field list for each data view. Note which ECS groups it fills and whether native names appear. Then run one search on host.name and see which sources it reaches.

• CuckooSkills: Log analysisTelemetry: Windows
• SignetSkills: Log analysisTelemetry: Linux · Network
• TrailheadSkills: Log analysisTelemetry: Linux · Network

References

• Elastic Common Schema reference · Elastic
• ECS field reference · Elastic
• ECS host fields · Elastic