Ep 023: Poster Child

Nate gets messy finding ingredients for his algorithm cake.

• Last week we focused on how to determine what to post.
• This week we focus on getting the data so we can compare it.
• (01:55) Once again, we'll use component to organize our app.
• What components should we have?
• Component 1: The worker that wakes up, checks the DB, checks Twitter, and posts as necessary.
• Debate: Do we need more than one component?
• Question: What does that worker need? Those should be components.
• (03:00) Component 2: The database connection.
  • Needs to be threadsafe
  • Allows all the DB logic in one place.
  • start method is a natural place to do migrations, indexing, etc.
• (04:45) Aside: quick component refresher
  • Very lightweight mechanism for shared, stateful resources.
  • Dependency injection: components depend on other components, but framework instantiates them.
  • Each component has two lifecycle methods start and stop.
  • Setup state in start: initialize internal data, open connection, prep work, etc.
  • Tear it all down in stop: finishing work, closing connections, etc.
  • Declare all your components in a system.
• (06:00) What should we call Component 1? "Poster"?
• "It could be the poster child of components!"
• (06:50) Component 3: The Twitter API handle
• What is a "Twitter API handle" (aka the "Twitter handle")?
  • Data structure and code that handles authenticating and making requests.
  • Not a long standing socket connect (like the DB), but there is still state (auth key).
• (08:50) What happens when the cached credentials expire?
• Two options from Ep 006:
  1. Request function returns a tuple: [updated-handle, result]. The handle will only change when it has to re-auth.
  2. Use an atom for the handle. Request function mutates the handle when it has to re-auth.
• In both cases, the Twitter wrapper deals with re-auth. The difference is whether the code that uses the handle needs to know about it.
• We'll use #2 for our Twitter component.
• We must still consider a race condition between components who all use an expired handle at the same time. Extra work will be done, but the result will still be correct.
• If components use separate handles, they can't benefit from sharing the re-auth.
• (12:55) We need a way to trigger "waking up"
• Use at-at to schedule an interval for calling a function
• Backed by Java's ScheduledThreadPoolExecutor.
• "It's good to have your components be tidy and clean up after themselves."
• Important to call at-at/stop in your component/stop method so component.repl/reset doesn't make more and more timers!
• (16:10) We have the ingredients to implement the algorithm
• Basic steps:
  • Wrap it all in an exception handler
  • Use a sequence in a let block to assign results back from Twitter and DB
  • Take results and pass them to pure function to determine what to do next
• Complication: order dependent. We need last posted Tweet to know how far back to fetch from Twitter
• New steps:
  • Fetch from DB: last posted and next scheduled
  • Use last posted ID to fetch from Twitter
  • Pass tweets and next scheduled tweet to pure function to determine if we should post
  • If we need to post, try posting to Twitter and capture result
  • If success, write tweet ID into database to mark as "completed"
  • No matter what, write the result to the "attempt" log
• (22:00) Feels very imperative
• "Do I/O. Do some logic. Do I/O. Do some logic. All those little bits of logic are very hard to test."
• OO answer: mock the resources
• Mocking makes more problems: now you have to implement all sorts of fake logic
• "You just start grabbing more and more side effects and glomming them on to this big ball of mud, just so you can test a little bit of logic."
• "Next thing you know, you're developing a whole vocabulary of mock creation."
• We'll look at this more next week.

Message Queue discussion:

• (25:00) We were mentioned on the Illegal Argument podcast
• Comparing Clojure REPL and Smalltalk REPL.
• Common problem: state in the REPL does not reflect what is in the source.
• For us, connected editor helps avoid that: we run pieces directly from our source files.
• Still can end up out of sync: dangling symbol references.
• Using tools.namespace.repl/refresh will find those dangling references.
• Can still build up in pieces, but can use refresh to check it all at once.

Related episodes:

• Twitter handle, retrying on failure
  • 006: All Wrapped Up in Twitter

Related projects:

• Component
• component.repl
• At-At
• Java ScheduledThreadPoolExecutor
• tools.namespace

Clojure in this episode:

• atom
• component/
  • start
  • stop
  • system-map
• component.repl/
  • reset
• at-at/
  • mk-pool
  • every
  • stop