Ep 033: Cake or Ice Cream? Yes!

Nate needs to parse two different errors and takes some time to compose himself.

• Previously, we were able to parse out errors and give the parsing function the ability to search as far into the future as necessary.
• We did this by having the function take a sequence and return a sequence, managed by lazy-seq.
• (01:30) New Problem: We need to correlate two different kinds of errors.
• The developers looked at our list of sprinkle errors and they think that they're caused by the 357 errors.
• They have requested that we look at the entire log and generate a report of 357 and sprinkle errors, so we can tell if they're correlated.
• "When someone says, do I want cake or ice cream, the right answer is: yes, I want both!"
• Before, we were only parsing out a single type of error and summarizing it, but now we need to parse out both types of errors.
• If we try to parse both kinds of errors with the same function, we will quickly get ourselves into nested ifs or maybe an infinite cond. Perhaps a complex state machine with backtracking?
• (05:55) Realization: Each error stands alone. Once you detect the beginning of a sprinkle error, you won't need to look for a 357 error.
• You can take each one in turn.
• (06:30) Solution step 1: What if we had two functions, one for each type of error.
• Each of these functions would take the entire sequence and tell us if there was an error at the beginning.
• Previously, our function both recognized errors and handled the sequence generation. If we pull those apart, we can add parsing for more errors easily.
• Each error parsing function would return nil if no error was found at the head of the sequence.
• (08:46) Solution step 2: Create a function that uses the two detectors to find out what error is at the head of the sequence.
• It takes the sequence, and wraps consecutive calls in an or block.
• The or block will try each one in turn until one matches and then that is the result.
• Each error's parsing is in its own function, and the combining function serves as an inventory.
• (11:35) Solution step 3: Create a lazy sequence that wraps calls to the combined detector function.
• Last week's code has parsing and lazy in one function.
• Now that we've pulled the parsing out, we can use the remaining structure to create our lazy sequence.
• The combined detector function is parse-next, and the function that manages the lazy sequence is parse-all.
• "Now we've fulfilled our obligation to have bike-shedding on naming. Next up, cache consistency. And finally, off-by-one errors."
• The top of parse-all has a call to lazy-seq.
• It will use the result of calling parse-next on the sequence.
  • If it gets something, it will use cons to add that value to the beginning of a recursive call to itself.
  • If it gets nil, it will recursively call itself with the rest of the sequence, thus advancing the parsing forward one step.
• It's not a ton of boilerplate, but it is nice to put all the mechanics of the sequence creation into a function by itself.
• Now we have a heterogeneous sequence of errors, and we can transform it into any report that is useful.
• Each parsing function doesn't need to worry about advancing down the sequence, that is handled by the higher parse-all function.
• Since we have a new lazy sequence, we can take it and make recognizers that take it and generate an even higher level of sequence.
• We ruminate more on higher level data in Episode 020.

Related episodes:

• 020: Data Dessert
• 028: Fail Donut
• 029: Problem Unknown: Log Lines
• 030: Lazy Does It
• 031: Eager Abstraction
• 032: Call Me Lazy

Clojure in this episode:

• seq, cons, rest
• lazy-seq
• or, cond

Code sample from this episode:

(ns devops.week-05
  (:require
    [devops.week-01 :refer [parse-line]]
    [devops.week-02 :refer [process-log]]
    [devops.week-03 :refer [sprinkle-errors-by-type]]
    ))

(defn parse-sprinkle
  [lines]
  (let [[first-line second-line] lines
        [_whole donut-id] (some->> first-line :log/message (re-matches #"failed to add sprinkle to donut (\d+)"))
        [_whole error] (some->> second-line :log/message (re-matches #"sprinkle fail reason: (.*)"))]
    (when (and donut-id error)
      (merge first-line
             {:kind :sprinkle
              :sprinkle/donut-id donut-id
              :sprinkle/error error}))))

(defn parse-357-error
  [lines]
  (let [[first-line] lines
        [_whole user] (some->> first-line :log/message (re-matches #"transaction failed while updating user ([^:]+): code 357"))]
    (when user
      (merge first-line
             {:kind :code-357
              :code-357/user user}))))

(defn parse-next
  [lines]
  (or (parse-357-error lines)
      (parse-sprinkle lines)))

(defn parse-all
  [lines]
  (lazy-seq
    (when (seq lines)
      (if-some [found (parse-next lines)]
        (cons found (parse-all (rest lines)))
        (parse-all (rest lines))))))

(defn kind?
  ([kind]
   #(kind? % kind))
  ([line kind]
   (= kind (:kind line))))


(comment
  (process-log "sample.log" #(->> % (map parse-line) parse-all (map :kind) doall))
  (process-log "sample.log" #(->> % (map parse-line) parse-all (filter (kind? :sprinkle)) sprinkle-errors-by-type))
  )

Log file sample:

2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | transaction failed while updating user joe: code 357
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | failed to add sprinkle to donut 23948
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | sprinkle fail reason: should never happen
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | failed to add sprinkle to donut 94238
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | sprinkle fail reason: timeout exceeded threshold
2019-05-14 16:48:56 | process-Poster | INFO  | com.donutgram.poster | transaction failed while updating user sally: code 357
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | failed to add sprinkle to donut 24839
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | sprinkle fail reason: too many requests
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | failed to add sprinkle to donut 19238
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | sprinkle fail reason: should never happen
2019-05-14 16:48:57 | process-Poster | INFO  | com.donutgram.poster | transaction failed while updating user joe: code 357
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | failed to add sprinkle to donut 50493
2019-05-14 16:48:55 | process-Poster | INFO  | com.donutgram.poster | sprinkle fail reason: unknown state