Log sharing with RAW

At RAW, our development and operational teams often need to work together to debug those harder bugs. Sometimes our standard pipelines don't quite capture all the necessary information and we need to dig a bit deeper. This means Ops sharing logs with Devs. But how to do it securely?

So inspired by our own needs, here's a simple example on how to accomplish that with RAW.

Say you want to share data from a log file with others. But you cannot share sensitive information, and you also need to limit the data displayed depending on the user or application calling the endpoint.

To limit access in the RAW platform, we can use scopes. Scopes are a list of strings that you define and that can be assigned to users or API keys. So you can create a scope called "monitoring" and assigned it to a collection of users and API keys; this scope can then be used to limit the access to the API; for more information on this, check our API Key management documentation page.

For our example, each log file has 3 levels of log statements; INFO, WARN and ERROR. We want to implement the following data access rules:

• Users with the scope admin scope can see all statements.
• Users with the scope monitoring can see INFO and WARN statements.
• Users without any relevant scopes can only see INFO statements.

Here's an excerpt of the log file:

2015-01-01T05:54:15 WARN vibration close to treshold, check instrumentation panel ASAP.
2015-01-01T05:54:58 INFO calibration at 100%, checking inner sub-systems.
2015-01-01T05:55:41 ERROR voltage not measured for more than 25 seconds, reboot machine.
2015-01-01T05:56:24 INFO cleaning procedure schedulled soon, performing sub task 111.
2015-01-01T05:57:07 INFO task 155 schedulled soon, preparing next task.

Parsing the file

In Snapi, we use String.ReadLines to split the file into lines of text and Regex.Groups to extract the timestamp, level and message from each line. Like this:

parse() =
    let
        lines = String.ReadLines(
            "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
        ),
        parsed = Collection.Transform(lines, l ->
            let
                groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
                timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
                level = List.Get(groups, 1),
                message = List.Get(groups, 2)
            in
                {timestamp: timestamp, level: level, message: message}
        )
    in
        parsed

Note

We are using a regex pattern to parse the file. Learn more about regex patterns here.

The output looks like:

[
  {
    "timestamp": "2015-01-01T05:54:15.000",
    "level": "WARN",
    "message": "vibration close to treshold, check instrumentation panel ASAP."
  },
  {
    "timestamp": "2015-01-01T05:54:58.000",
    "level": "INFO",
    "message": "calibration at 100%, checking inner sub-systems."
  },
  {
    "timestamp": "2015-01-01T05:55:41.000",
    "level": "ERROR",
    "message": "voltage not measured for more than 25 seconds, reboot machine."
  },
  {
    "timestamp": "2015-01-01T05:56:24.000",
    "level": "INFO",
    "message": "cleaning procedure schedulled soon, performing sub task 111."
  },
  {
    "timestamp": "2015-01-01T05:57:07.000",
    "level": "INFO",
    "message": "task 155 schedulled soon, preparing next task."
  }
]

Filtering by scope

Now that we have parsed the file we can start implementing our data restriction rules. We will use the built-in function Environment.Scopes to get the caller scopes and filter the data accordingly. Something like this:

Collection.Filter(
    parsed,
    (x) ->
        if List.Contains(Environment.Scopes(), "admin")
            then true
        else if List.Contains(Environment.Scopes(), "monitoring")
            then x.level == "WARN" or x.level == "INFO"
        else
            x.level == "INFO"
)

Here is all the code together:

main() =
    let
        lines = String.ReadLines(
            "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
        ),
        parsed = Collection.Transform(
            lines,
            (l) ->
                let
                    groups = Regex.Groups(
                        l,
                        "(\\d+-\\d+-\\d+T\\d+:\\d+:\\d+) (\\w+) (.*)"
                    ),
                    timestamp = Timestamp.Parse(
                        List.Get(groups, 0),
                        "yyyy-M-d\'T\'H:m:s"
                    ),
                    level = List.Get(groups, 1),
                    message = List.Get(groups, 2)
                in
                    {timestamp: timestamp, level: level, message: message}
        )
    in
        Collection.Filter(
            parsed,
            (x) ->
                if List.Contains(Environment.Scopes(), "admin")
                    then true
                else if List.Contains(Environment.Scopes(), "monitoring")
                    then x.level == "WARN" or x.level == "INFO"
                else
                    x.level == "INFO"
        )

Using API keys

API keys provide a secure way to give access to private endpoints to users without the necessary permissions (scopes) to access them.

We can create API keys with the scope monitoring or admin, and set the expiration time e.g. one day. We can now give this key to users, so they can see ERROR or WARN messages, and not even having to worry about revoking access, as it will happen by itself.

That's a much safer way to share logs with others!

For access to fully-working ready-to-use example, check our related demo data product here.