Ep 103: Explorify!

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to [email protected], or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "exploring new data and APIs". We peruse APIs to uncover the data hidden beneath.

Our discussion includes:

• More exploration of Sportify!
• How to explore a database from the REPL.
• How to work with SQL rapidly.
• How structural editing speeds things up.
• Building up the vocabulary of exploration.
• What should the process of exploring via the REPL feel like?
• What is a Media Asset Manager?
• How do you make sense of a brand new JSON API you've never worked with?
• How do you work with a poorly documented API?
• What problems are complected for HTTP requests?
• How to use composition to speed up API exploration.
• Dealing with the non-linear nature of exploration.
• Saving results and experimenting with them.
• How can you speed up experimentation on large data sets?
• How to handle excessively normalized APIs.

Selected quotes

• "This is a situated problem. It's a real-world problem. Like many real-world problems, there are parts we control and parts we don't control. We have to figure out those parts."

• "What do we need to know to figure out the information?"

• "I like to start with the way people naturally talk about these things. If you start building up your language, it helps to describe things in either the innate input the system must have or the way people talk about it."

• "[Honey SQL] will do all the interpolation for you. It's just wonderful! That one feature alone would be enough, but there's plenty more."

• "That's a great thing about Hiccup and Honey SQL and other formats that use Clojure data structures to describe things: you get the power of structural editing."

• "It's yet another example of building up the vocabulary of the system so that you're able to talk about it at a higher level."

• "Practicality is the name of the game here. Imagine you're exploring in a forest and you're trying to figure out what you want to put in your backpack to take along with you. You don't want to take a lot of heavy, complicated tools. You want to only bring along the stuff that is actually useful to you!"

• "You only need to build up what is necessary to keep exploring because that's the point. The point is to learn. The point isn't to pre-optimize and make the abstractions."

• "My activity is centered around this fiddle file as I'm exploring."

• "We have all this cool information, but we're not making database table highlight reels. We're making sports highlight reels in Sportify!"

• "How do you make sense of a brand new JSON API that you have never dealt with before?" "By using it."

• "This is a good opportunity to mention we have a series called 'Web of Complexity'. The title should give you a sense of what we think of HTTP in general."

• "The name web should already be a bad omen! We never use the name 'web' in a positive sense anywhere else."

• "Perfect time to use our nonlinear history, aka the fiddle!"

• "We've made several really composable ingredients that we're now mixing together as we're learning more about the system."

• "Most of my time was spent sifting through the data, not actually making the calls!"

• "I'm communicating to myself tomorrow, because I want to forget all this context, go home, do something else, not think about work, and come back to work and pick it all up again."

• "One of the benefits of using Clojure to explore is you have a full, rich programming language—one with a full syntax including comments. By doing it all in one fiddle with these comments, you can pick it up in the morning."

• "Our lives are nonlinear. We get interrupted. We take a break. We have the audacity to go home and not think about work!"

• "It's like a workbench. We're laying out our tools. We're laying out our pieces. Our fiddle file is our workbench and we leave little post-it notes on that workbench to remind us of things."

Example code

Here is an example that uses the Cloudflare Streams API. It requires authentication, so we want to factor that out.

First, define some endpoints to work with. Create pure functions for just the part unique to each endpoint.

(defn check-status-req [id]
  {:method :get
   :path id})

(defn delete-req [id]
  {:method :delete
   :path id})

Then create a function to expand the request with the "common" parts:

(defn full-req
  [cloudflare api-req]
  (let [{:keys [api-key account-id]} cloudflare
        {:keys [method path], throw? :throw} api-req]
    {:async true
     :method method
     :uri (format "https://api.cloudflare.com/client/v4/accounts/%s/stream/%s" account-id path)
     :headers {"Authorization" (str "Bearer " api-key)
               "Accept" "application/json"}
     :throw throw?}))

For the purposes of illustration, the cloudflare configuration is:

(def cloudflare
  {:api-key "super-secret"
   :account-id "42424242"})

See the full request like so:

(full-req cloudflare (check-status-req "abcdefg123456789"))

Which returns:

{:async true
 :method :get
 :uri "https://api.cloudflare.com/client/v4/accounts/42424242/stream/abcdefg123456789"
 :headers {"Authorization" "Bearer super-secret"
           "Accept" "application/json"}
 :throw nil}

Use [babashka.http-client :as http], and call the endpoints like so:

@(http/request (full-req cloudflare (check-status-req "abcdefg123456789")))
@(http/request (full-req cloudflare (delete-req "abcdefg123456789")))

Note, that in a REPL-connected editor, you evaluate each form, so you can see just the check-status-req part, or move out one level and see the full-req part, or move out one more level and actually run it. That lets you iron out the details to make sure you have them right.

Finally, you can make some helpers to use in the REPL or imperative code. The helpers stitch the process together:

(defn request!
  [cloudflare req]
  (-> @(http/request (full-req cloudflare req))
      (update :body #(json/parse-string % true))))

(defn check-status!
  [cloudflare id]
  (request! cloudflare (check-status-req id)))

(defn delete-stream!
  [cloudflare id]
  (request! cloudflare (delete-req id)))

They can be called like:

(check-status! cloudflare "abcdefg123456789")
(delete-stream! cloudflare "abcdefg123456789")

The helpers should do nothing except compose together other parts for convenience. All the real work is in the pure functions.

Links

• Ep 014: Fiddle with the REPL
• Ep 063: Web of Complexity
• Sportify! series:
  • Ep 101: Sportify!
  • Ep 102: REPLify!
• next.jdbc
• Honey SQL
• Babashka http-client
• clj-http
• http-kit
• Portal
• Nippy