Sveriges mest populära poddar

Functional Design in Clojure

Ep 028: Fail Donut

17 min • 10 maj 2019

Christoph has gigs of log data and he's looking to Clojure for some help.

  • Introducing a new topic.
  • The last few weeks we focused on Twitter and automatically posting to it.
  • Surprised by how much there is to talk about in a focused problem.
  • "There will always be more problems for the world to solve."
  • (01:53) Imagine if you will, the world of DonutGram!
  • A fictitious social network where people post their donut experiences.
  • This series is not about making DonutGram, but living DonutGram.
  • "Oh, the dark underbelly of application development: support."
  • "Let's take the shiny rock and flip it over and look at all the worms and bugs."
  • We talked about forensic data before, and much of DevOps is about looking at that data.
  • You want to paint the most complete story. It's a development problem too.
  • Most of the time, forensic data is written to a log file. It's the first line of investigation.
  • Imagine all of the components of your application speaking into one pipe, and you now get to reconstruct what happened.
  • (05:59) Everything is rosy at DonutGram, but then users start having issues.
  • Users get the "Fail Donut" and start posting that.
  • We will be assuming the role of the heroic DevOps team.
  • There is quite a bit of data written to the log file, more and more with each bugfix.
  • We're going to use Clojure to investigate this problem.
  • (08:50) What problems might we encounter?
  • Problem 1: Size
    • No one configured log rotation!
    • The log file is too large to load for analysis.
  • Problem 2: Unstructured data
    • Everything is a line or multiple lines.
    • We can build up layers of abstractions to gradually build understanding.
  • Problem 3: Non-linear data
    • We want to tell a story about what went wrong.
    • Many times, the pieces of that story are in different areas of the log file, and must be collated.
    • There can be multiple competing stories.
    • The parts of the story are like dominoes. How do you know if the third domino is important without knowing that the first two have fallen?
  • Problem 4: Alerts
    • "If you have a really good story to tell, how do you tell anyone about it?"
    • Alerts should be timely, depending on the audience.
  • Call out to the audience: do you have battle stories about processing log files? Let us know!

Clojure in this episode:

  • nil
Förekommer på
00:00 -00:00