Jekyll2021-05-11T15:35:44-04:00https://casavant.org/feed.xmlJeff CasavantMy personal blog
Good metrics2020-10-21T18:37:00-04:002020-10-21T18:37:00-04:00https://casavant.org/2020/10/21/metrics<p>What makes a good metric?</p>
<!--more-->
<p>When I say <q>metric</q> for the purposes of this post, I mean something specific.
I don't mean every bit of data you can get, every rollup and statistical trick
you can do to produce an interesting number. I mean things that will be
included in dashboards, used to evaluate performance, or used to drive alerting
and scaling decisions. By <q>metrics</q> I mean to call out <em>practical</em> numbers,
whether human or machine facing.</p>
<p>As for what <em>makes</em> a metric good, I'd point to three main factors.</p>
<h2>Intuitive</h2>
<ul>
<li>Metrics should be trivial to understand. It should not require additional
thought or mental combination with other metrics to come to a conclusion.
You should not be able to come to opposite conclusions from the same metric.</li>
<li>Metrics should take advantage of automatic mental processes and not ignore or
contravene them. The correct conclusion from a metric should never be the
opposite of the initial impression.</li>
</ul>
<p>Some examples of unintuitive metrics:</p>
<ul>
<li><strong>CPU Usage.</strong> 98% CPU usage seems like a bad thing: the server is almost at
capacity. The natural conclusion is that we need to scale up or out to
alleviate the pressure. Working from only this metric, you'll probably end
up with many of your servers sitting at 80% CPU usage since that feels more
comfortable. That's a bad outcome, though: with a constant workload, you're
paying for 25% more CPU capacity than you use. With a burst-style workload,
it makes sense to overprovision CPU. This metric could therefore be improved
by bringing in the knowledge of what workloads your servers typically operate
at. Track a high water mark for burst (e.g. 2nd standard deviation or 95th
percentile) and make your server provisioning decisions based on that.</li>
<li><strong>Error count.</strong> An increase in the number of errors could indicate a major
problem but it could just indicate higher than normal usage.</li>
</ul>
<h2>Actionable</h2>
<ul>
<li>Metrics should clearly motivate decisions. It's alright if that decision is
to stay the course - with metrics backing it, you can be confident it is the
right decision. Metrics should not indicate multiple courses of action
depending on interpretation.</li>
</ul>
<p>Some examples of unactionable metrics:</p>
<ul>
<li><strong>Lines of code.</strong> Is a developer who ships more code more productive or more
likely to burn out? Or is their high line count because they use more
verbose languages or supporting tools? It's completely unclear what to do
once you have this number. We could improve it by bucketing it. The range
of normal code contributions is probably quite wide, but we should be able to
establish a lower bound for acceptable contributions in a given timeframe for
a developer. For your organization this may be a single line, leaving us
with a bucket for 0 lines and a bucket for any contribution at all - but
we've made the metric actionable (assuming we are also able to identify which
employees need to be slinging code at all). It's not incredibly useful, but
if we were to hire a dev who did absolutely nothing, this metric would
motivate some action.</li>
<li><strong>Test coverage.</strong> Test coverage is another that can be made useful by
bucketing. If coverage right now is 95%, who's to say that last 5% is worth
adding? And who would commit to saying that the test coverage should be no
<em>lower</em> than it is now? But if we again set a remote lower bound - say 70% -
falling below that should clearly trigger a re-evaluation of the test setup.</li>
</ul>
<p>Some examples of actionable metrics:</p>
<ul>
<li><strong>Percent of files touched in the average pull request.</strong> If this is high,
that clearly indicates a need for refactoring, since there's a lot of
interdependence in the project.</li>
</ul>
<h2>Complete</h2>
<ul>
<li>Metrics should bring in all relevant information. Metrics should not ignore
part of the problem. This does not mean that metrics need to incorporate all
possible indicators - just that everything relevant should be represented in
the final metric. It's entirely acceptable to include only one system usage
indicator to build a metric that drives decisions about scaling, but skipping
that and just using job count on a particular server will fall short of the
mark.</li>
<li>It's part of the above, but I want to call it out specifically: metrics
should be up to date. Quickly incorporating recent information is important.</li>
</ul>
<p>Some examples of incomplete metrics:</p>
<ul>
<li><strong>Badly-formatted packets entering a network.</strong> This could be used to
indicate a possible attack but it needs a bit more in order to be complete.
We should combine it with a comparison against the signatures of known
hacking tools (this sort of metric is available in your average intrusion
detection system).</li>
</ul>
<p>I'm sure I've missed something here. This is the product of only a few hours'
thought. I'm interested to hear what good & bad metrics you have encountered -
do the bad ones fail for one of these reasons? Drop a comment below.</p>What makes a good metric?Wireguard gotchas with multiple tunnels2020-10-10T10:42:00-04:002020-10-10T10:42:00-04:00https://casavant.org/2020/10/10/wireguard-fwmark<p>Wireguard has a bit of a gotcha when running multiple independent tunnels, one
of which has a default route associated with it.</p>
<!--more-->
<h1>Symptoms</h1>
<p>I have two Wireguard tunnels on my system. One has a default route, and the
other does not. Ping latency is surprisingly high on the tunnel without the
default route.</p>
<h1>Solution</h1>
<p>Add a matching <code>FwMark</code> setting to your non-default-route tunnel. Typically
this will be <code>FwMark = 51820</code>, but check the logs or <code>Table</code> setting of your
default route tunnel to see what mark it's using.</p>
<h1>Technical deep dive</h1>
<p>Let's start with some background. Imagine you have the following routing table
on your system:</p>
<div class="highlight"><pre><code class="language-" data-lang="">$ ip route
0.0.0.0/0 via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.2 metric 302
192.168.1.0/24 dev eth0 proto dhcp scope link src 192.168.1.2 metric 302
</code></pre></div>
<p>Routing decisions are made by matching the most specific route - so from the
bottom of the table upwards.</p>
<p>When OpenVPN is set up to route all traffic, it will insert an extra couple of
more specific routes to make sure no traffic exits via the raw interface
(except the transport traffic).</p>
<p>At this point I'm going to be making these routes up - they should be very
close, but the syntax may not be correct or complete. They'll work to
illustrate this problem.</p>
<div class="highlight"><pre><code class="language-" data-lang="">$ ip route
0.0.0.0/0 via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.2 metric 302
0.0.0.0/1 via <server tunnel IP> dev tun0
128.0.0.0/1 via <server tunnel IP> dev tun0
192.168.1.0/24 dev eth0 proto dhcp scope link src 192.168.1.2 metric 302
<server internet IP> via 192.168.1.1
</code></pre></div>
<p>Note the addition of two routes that cover half the IPv4 space each. This way,
they're more specific than the default route (<code>0.0.0.0/0</code>) but still cover the
same IP space. The last piece is the very specific route to just the VPN
server, which will match the transport traffic.</p>
<p>Let's do the same with a Wireguard tunnel that will route all traffic.</p>
<p>First, the configuration we'll feed into <code>wg-quick</code>.</p>
<div class="highlight"><pre><code class="language-" data-lang="">[Interface]
PrivateKey = <private key>
Address = <client tunnel IP>
[Peer]
PublicKey = <server public key>
AllowedIPs = 0.0.0.0/0
Endpoint = <server internet IP>
</code></pre></div>
<p>Then let's inspect our routing table:</p>
<div class="highlight"><pre><code class="language-" data-lang="">$ ip route
0.0.0.0/0 via 192.168.1.1 dev eth0 proto dhcp src 192.168.1.2 metric 302
192.168.1.0/24 dev eth0 proto dhcp scope link src 192.168.1.2 metric 302
</code></pre></div>
<p>No routes to our Wireguard server! What happened?</p>
<p>The answer lies in how Wireguard solves the problem of routing the transport
traffic. I would have expected it would do something similar to OpenVPN
(creating routes with specificity such that they end up in the correct order in
the routing table). It clearly doesn't do that, though. Let's check out
<code>wg-quick</code>'s logs and see what it did.</p>
<div class="highlight"><pre><code class="language-" data-lang=""># wg-quick up wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add <client tunnel IP> dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] wg set wg1 fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] sysctl -q net.ipv4.conf.all.src_valid_mark=1
[#] iptables-restore -nV
</code></pre></div>
<p>Taking a look at the route add command specifically, you can see it's doing
something a bit odd: it's creating a new route <em>table</em> with the id <code>51820</code>.
Let's print that table's contents.</p>
<div class="highlight"><pre><code class="language-" data-lang="">$ ip route show table 51820
0.0.0.0/0 via <openvpn tunnel IP> dev wg0
</code></pre></div>
<p>There it is! That's the default route we were missing before. The next piece
is how traffic ends up routed with <em>this</em> table as opposed to the main table.
You can see this being set up in the line that follows the default route
creation in the log:</p>
<div class="highlight"><pre><code class="language-" data-lang="">[#] ip -4 rule add not fwmark 51820 table 51820
</code></pre></div>
<p>This uses <code>iptables</code>' packet marking functionality to say any packet that is
<em>not</em> marked 51820 should use the alternate routing table (and thereby go out
the VPN tunnel rather than any other interface on the box). The last piece
we've got to handle is making Wireguard's transport traffic use the main
routing table rather than going out its own interface. Wireguard has the
<code>fwmark</code> setting for this purpose.</p>
<div class="highlight"><pre><code class="language-" data-lang="">[#] wg set wg1 fwmark 51820
</code></pre></div>
<p>So this combination of settings causes your system to route all traffic except
wg0's transport traffic out over wg0.</p>
<p>What happens if we add a second Wireguard tunnel with the following
configuration?</p>
<div class="highlight"><pre><code class="language-" data-lang="">[Interface]
PrivateKey = <private key>
Address = <client tunnel IP>
[Peer]
PublicKey = <server public key>
AllowedIPs = 192.168.1.0/24
Endpoint = <server internet IP>
</code></pre></div>
<p>It's kind of subtle, and you might not expect it if you haven't done policy
based routing with <code>iptables</code> before. The only difference between this tunnel
and the other is the <code>AllowedIPs</code> configuration. For this tunnel, we're only
routing a specific remote network. This means we won't have to solve the
transport traffic routing problem - it'll just work, since our VPN server is
not inside of the <code>AllowedIPs</code> range. Which means that <code>wg-quick</code> won't
attempt to solve that problem. Let's take a look at the logs.</p>
<div class="highlight"><pre><code class="language-" data-lang=""># wg-quick up wg1
[#] ip link add wg1 type wireguard
[#] wg setconf wg1 /dev/fd/63
[#] ip -4 address add <client tunnel IP> dev wg1
[#] ip link set mtu 1420 up dev wg1
</code></pre></div>
<p>It didn't do any of the alternate routing table stuff it did before. In
particular, it did not set Wireguard's <code>fwmark</code>.</p>
<p>See it now?</p>
<p>wg1 ends up nested inside of wg0. Since wg1's transport traffic is <em>not</em>
marked, it'll get put on the alternate routing table, and ship out via wg0.
The solution is pretty simple. We only need to add <code>FwMark</code> to the
configuration.</p>
<div class="highlight"><pre><code class="language-" data-lang="">[Interface]
PrivateKey = <private key>
Address = <client tunnel IP>
FwMark = 51820
[Peer]
PublicKey = <server public key>
AllowedIPs = 192.168.1.0/24
Endpoint = <server internet IP>
</code></pre></div>
<p>This makes wg1's transport traffic use the main routing table since it is now
marked.</p>Wireguard has a bit of a gotcha when running multiple independent tunnels, one of which has a default route associated with it.2018 in review2018-12-31T14:28:00-05:002018-12-31T14:28:00-05:00https://casavant.org/opinion/2018/12/31/review<p>As this year of blog posts comes to an end, I'd like to look back at what I wrote this year.</p>
<!--more-->
<p>I started off with some more well-prepared stuff. My post on <a href="https://jeffcasavant.com/advertising/2018/01/06/advertising-signaling.html">advertising as signaling</a> took me a few hours to write, but my more recent posts have taken 30-45 minutes. The length is comparable (the advertising post being rated as a 6 minute read but most of my recent posts are 4 minute reads). I wrote a few other pieces detailing some thoughts I had (including <a href="https://jeffcasavant.com/opinion/2018/01/13/long-term-concerns-programming.html">programming as a business</a> and the <a href="https://jeffcasavant.com/social/2018/05/06/clock-tower.html">student response to the OSU clock tower</a>). All of those pieces were fun to write, and they got easier as I became more practiced. I have a few filler pieces in there as well (delivering weekly was a bit of a chore for me).</p>
<p>The piece I'm most proud of is probably <a href="https://jeffcasavant.com/technical/2018/05/28/non-deterministic-tech.html">Choose the right abstraction</a>. I think this is this most interesting idea I wrote about - that an abstraction presented to a user is a promise to act in a certain way. It was fun to write and came directly from experiences I was having at the time. This is the kind of post I want to write more of. Another post that sticks out in my mind is <a href="https://jeffcasavant.com/misc/2018/09/24/mastodon.html">Mastodon: an update</a>, the post in which I detail why I ended up shutting down my Mastodon instance. I was probably going to do this anyways, but having the post to write that week made me crystallize the reasons why I was doing so. Writing it out helped me think clearer about it and I'm glad to have my thought process and reasons recorded here. Hopefully it's useful to someone else.</p>
<p>I put comments on my site a couple of months in (not sure exactly when). I've gotten participation mostly from my counterpart in the year-of-blog-posts bet (it occurs to me now I may not have mentioned the details of this all year - <a href="http://tcolmstead.com/">Taylor Olmstead</a> and I owe each other beers for each weekly post we missed in 2018). However, I did get some discussion with another friend of mine, <a href="https://maxwellbuck.com/">Max Buck</a>, and most unexpectedly from the author of a piece of software I wrote about (<a href="https://jeffcasavant.com/general/2018/07/23/kanban-mail.html">Kanban Mail</a>). The comments have been pretty solid overall; Disqus has done a great job of keeping out the spam.</p>
<p>I'm probably going to go through and take down a few of the posts I was clearly getting done last-minute. In the new year I'm not going to be held to the once-a-week schedule anymore, and will only be writing when I have something I want to write about. I expect by removing some of the posts I would not have made, I'll end up with a much higher quality end product that still has quite a bit of volume.</p>
<p>Overall, I have enjoyed writing so consistently. I think I've become better at the part I was most interested in improving - specifically, writing extemporaneously. I became better at having a much rougher outline in place and writing my posts from start to finish while maintaining a coherent train of thought (though you may think differently). I expect the practice I got will serve me well as long as I continue to exercise.</p>
<p>I'm looking forward to the next year.</p>As this year of blog posts comes to an end, I'd like to look back at what I wrote this year.Project idea: annotating political bias in internet articles2018-12-24T13:42:00-05:002018-12-24T13:42:00-05:00https://casavant.org/project/2018/12/24/project-idea-political-bias<p>Here's an idea: a browser addon that injects bias annotations into articles you're reading.</p>
<!--more-->
<p>I'd like to see a program that can annotate proper nouns with their political biases, or at least their funding sources, in the articles I'm reading online. For example, if you run across</p>
<blockquote>
<p>a recent study by the Cato Institute</p>
</blockquote>
<p>the addon would annotate it like this:</p>
<blockquote>
<p>a recent study by the (generally libertarian-leaning) Cato Institute</p>
</blockquote>
<p>or if you found</p>
<blockquote>
<p>the Brookings Institution found that</p>
</blockquote>
<p>you would get</p>
<blockquote>
<p>the Brookings Institution (receiving funding from the UAE and JP Morgan & Chase) found that</p>
</blockquote>
<p>This would make it easier to weight research findings based on the biases that they're expected to have.</p>
<p>Unfortunately, to make such a project work & be actually useful, we would have to identify those biases that affect the various think tanks, and this would be subject to the same issues that we already have - namely that we have blind spots when identifying our own biases. We could ostensibly crowd-source the bias identification in order to take advantage of the idea that most people won't be in agreement with a given political position and will therefore be able to clearly see the bias. The funding source idea at least partially alleviates this since the political viewpoints of individuals are easier to pick out - even though in practice we're just mapping the domain of think tanks to the domain of donors.</p>
<p>I'm not sure exactly how you'd source the required data (except for funding sources - they're public), but I would like to see the result.</p>Here's an idea: a browser addon that injects bias annotations into articles you're reading.The Ballad of Buster Scruggs: Russian Literature for the West2018-12-09T16:40:00-05:002018-12-09T16:40:00-05:00https://casavant.org/opinion/2018/12/09/scruggs<p>The Ballad of Buster Scruggs exemplifies the feeling the Coen brothers seem to go for: the same bleak, lonely realism found in Russian literature.</p>
<!--more-->
<p>One of my favorite Russian short stories is The Overcoat by Nikolai Gogol. In it, a young man saves his money for months to purchase a new overcoat. When he finally gets the coat, his coworkers throw him a party; on the way home he is robbed of his new coat. He spends the rest of the story freezing in the Russian winter, trying in vain to get the authorities to track down his coat. In the end he catches cold and dies, just hours before the magistrate decides he should get the man a new coat.</p>
<p>Buster Scruggs is a set of six short stories. In each, we are presented with people engaged in a struggle to better themselves or their situations (with the possible exception of the first and last, but I'll address those later). As they toil they are presented with difficulties that force them to choose between equally bleak and hopeless options, giving us bleak and hopeless results.</p>
<p>As part of my thoughts going forwards I'm going to address some of the plot points. Spoilers abound; go watch the movie before you read this so you get the full emotional experience of the movie - it's worth it.</p>
<p>Perhaps the most heartbreaking story presented in Ballad is the story of a young handicapped orator. He's escorted from town to town by a more capable, but less intelligent, companion. We see many times the beauty of his delivery as he recites everything from Ozymandias to the Gettysburg Address. Over the course of this story we also witness the crowds he draws dwindle and the coins he brings in lessen. He ends up killed by his traveling companion when the companion purchases a more-popular mathematically-capable chicken. This story hit me the most. The way the Coen brothers stress the value of the performer, showing the variety of content and delivery and especially the emotion with which he recites these pieces, causes us to appreciate his talent. And the agonizingly long murder scene is clear in its outcome from the first. We are forced to watch the companion stop the wagon, walk to the bridge, and test the drop with a rock. Finally we are spared with a cut to the wagon rolling quickly over a field, and given a confirming clip that shows us the inside of the wagon - now containing only a chicken.</p>
<p>Each of the stories in Ballad is presented within the turning pages of a book. We can read the first couple of sentences at the beginning, and the last few words at the end. The performer's story, Meal Ticket, ends with the following sentence, capturing the feeling of what was lost:</p>
<blockquote>
<p>And so they traveled, forward, onward, toward dramas the outlines of which blurred in the dozing of the man, and were by the chicken dimly surmised.</p>
</blockquote>
<p>This story parallels in my mind the story of The Overcoat. We have in both cases a shining example of humanity: in The Overcoat, a man who works hard to improve his station, and achieves it; in Meal Ticket, a man who possesses an exceptional mind and exceptional talent, and with it brings high intellectual ideas to the common man. In both stories the conclusion we are forced to draw is that the world does not value the highest human ideals. The world is colder, more base, and the effort one puts into life is worth less than the station one was born into.</p>
<p>This feeling, though perhaps not this specific moral, is carried in the other stories as well. From the bank robber's last glimpse of beauty from the gallows in Near Algodones to the ever-so-slightly early suicide of the young girl in The Gal Who Got Rattled, we witness the stark contrast between what is good and what we are forced to deal with in life.</p>
<p>There are two stories that don't fit this mold in Ballad, however. First, there is the story of Buster Scruggs himself, a singer/gunslinger who is making his way across the desert. This story is more entertaining and surreal than anything else. I don't find it bleak or sad in any way; it's just an unusual take on the gunslinger story trope. The last story comes much closer to fitting into the Russian-literature archetype. We see five strangers on a stagecoach ride. They discuss many weighty topics ranging from marital fidelity to the nature of death, and most depart the stagecoach discomforted by their conversation. We do see this sort of short story in Russian literature. Specifically, there is a story called The First-class Passenger, by Anton Chekhov. In it two men on a train discuss the nature of fame, and how it doesn't attach itself to those who are truly deserving. While they end up laughing rather than uncomfortable, they are no less affected by their philosophical conversation.</p>
<p>The last thing I'd like to address is a Coen brothers staple. While you often hear that you should show rather than tell, most media will spoon-feed a story to you to make sure you've got it. This is not so in the Coen brothers movies I've seen, and Ballad remains focused on showing you the story and making you figure things out. As a result the story feels more genuine and causes more emotional response. It's a mark of high quality in film.</p>
<p>Overall, I enjoyed The Ballad of Buster Scruggs. It's a well done movie, with interesting music, good writing, and good camera work. Let me know what you thought & felt in the comments.</p>The Ballad of Buster Scruggs exemplifies the feeling the Coen brothers seem to go for: the same bleak, lonely realism found in Russian literature.On Bond2018-11-19T14:36:00-05:002018-11-19T14:36:00-05:00https://casavant.org/opinion/2018/11/19/on-bond<p>I watched the first Bond movie, Dr. No, this weekend & have some thoughts on James Bond as a cultural icon.</p>
<!--more-->
<p>Dr. No is the first James Bond movie. It premiered in 1962 and was quickly followed by three more movies, one a year until 1965. 1966 did not see a Bond movie, but they've been made consistently every few years since then, for a total of 26. As of now that's an average of one every two years. That's a remarkable feat for a movie franchise - especially one so consistent in its formula. James Bond is older than the moon landing, JFK's assassination, the Vietnam war, the fall of the Berlin Wall and the end of the cold war, the creation of the Internet, and cable TV. The consistency of the Bond formula through all that change makes the Bond films an interesting study into changing culture over time. I am going to ignore the questions of racism and sexism - I think quite and lot has been written connecting James Bond with those topics - and focus more on the trends of the series itself. Specifically I am interested in what sorts of things are established as the core features of Bond movies (at least for the early few, as they are introduced). I am also interested in how the characteristics of Bond villains change over time. I think this will have the most to say about what we were worried about as a culture at the time.</p>
<p>Let's start with Dr. No. Be warned, there are spoilers here.</p>
<p>The movie is set mainly in Jamaica (the end credits reveal that at least some portion was shot on-location, something I expect to change over the series). We see the introduction of many Bond mainstays, including an attractive woman with a pun name (in this case Honey Ryder's name is used to great effect with the theme, Under the Mango Tree - <q>Under the mango tree, my honey...</q>) and a solid action movie quip (though I only noticed one). As the pursuer's car drives off a cliff and explodes, a bystander asks Bond what happened. He says <q>I guess they were going to a funeral.</q> Bond's car, by the way, is nothing remarkable - a Sunbeam Alpine, a two-seater British sports car more akin to an MGB than an Aston Martin DB-series. That has yet to be established. His gun, however, is firmly put in place (by his boss). In this scene Bond is relieved of his preferred Beretta (<q>Nice and light - in a lady's handbag! No stopping power.</q>) and given his standard Walther PPK. Finally, we see the introduction of Bond villains and the SPECTRE organization. Bond villains are cold and unfeeling and they gloat when Bond tries and fails to advance his own ends. They have large fortresses full of traps and confusing passages. They only want to have the world bow before them, and nothing else will do. They're intelligent and rich and just about to accomplish what they've set out to do.</p>
<p>Dr. No in particular is a half-Chinese, half-German man. He's a very intelligent engineer who once worked for Chinese organized crime, before stealing a bunch of money and establishing a convoluted fortress in Jamaica. From there, he uses some sort of a nuclear-powered machine that makes radio waves to <q>topple</q> American missiles and rockets. What does this say about our cultural fears? Well, the use of nuclear power - and the infusion of the swamp surrounding Dr. No's lair with radiation - is an obvious pick for what we were afraid of at the time. Nuclear power remains something we're inordinately worried about. At this time, I anticipate the bombs were more prevalent in people's minds. It had been under twenty years since they were dropped. This makes nuclear power an appropriate touchstone. Dr. No is half-Chinese - the fear here is Communism - and half-German, which perhaps reaches back again to WWII, though the case could be made that it's also a fear of Communism since half of Germany is run by Russia at this point in time. The Berlin Wall had arisen just a year previous. Lastly, Dr. No's plans to topple American rockets would have hit home in particular - JFK delivered his famous <q>to the moon</q> speech in mid-1961.</p>
<p>I did not expect to find so much in Dr. No that plays on cultural fears. I expect reading Bond villains as a proxy for cultural fears will continue to be interesting over the rest of the series.</p>
<p>Does this interest you? Did I miss something? Let me know in the comments. I'm gonna go watch From Russia with Love and have more for you next week.</p>I watched the first Bond movie, Dr. No, this weekend & have some thoughts on James Bond as a cultural icon.What Matrix needs2018-11-12T14:15:00-05:002018-11-12T14:15:00-05:00https://casavant.org/opinion/2018/11/12/matrix-needs<p>Matrix is pretty dang solid overall; there's still some room for improvement if you've got the drive and the time.</p>
<!--more-->
<ul>
<li>An alternate homeserver implementation. There's only one production-ready homeserver right now: <a href="https://github.com/matrix-org/synapse">Synapse</a>. The main downsides with Synapse have historically been performance-related, but that's constantly improving - they've significantly reduced memory and compute usage in the past few months of releases. There are two front-runners to fill this slot (probably due to the complexity of the Matrix standard). <a href="https://github.com/matrix-org/dendrite">Dendrite</a> is the official project, spearheaded by the Matrix project leads. Second, I'd put <a href="https://github.com/ruma/ruma">Ruma</a> - but it's a far off second, it hasn't seen a commit in about a year.</li>
<li>More bridges. Personally I notice the absence of a functional GroupMe bridge (there is <a href="https://github.com/matrix-hacks/matrix-puppet-groupme">a bridge available</a> but it won't run on my system, and I haven't taken the time to troubleshoot it yet). There are many other chat services out there. Ideally, bridges should be double-puppet style, requiring only my own personal authentication to both services, or appservice style, providing connectivity transparently between a Matrix room and another chat service's room. These are the most useful bridge styles.</li>
<li>Solid webhooks integration. There's <a href="https://github.com/turt2live/matrix-appservice-webhooks">matrix-appservice-webhooks</a> which fulfills most of the requirements I have, but the majority of the services I use that offer a webhook integration expect Slack to be on the other end. Appservice-webhooks doesn't quite conform to the Slack webhook style, and as a result incoming content from some services gets garbled (resulting in projects like <a href="https://github.com/ananace/ruby-grafana-matrix">ruby-grafana-matrix</a> to provide a webhook endpoint that properly displays Grafana alerts).</li>
<li>Variety in languages used. Almost all of the Matrix ecosystem services I run are Node.js applications. Matrix itself is not, and neither is the Hangouts bridge; they're both Python. But that's the extent of it. Ruma, mentioned above, is written in Rust; it's far from production-ready (and appears dead as a project). I'd like to see more bridges written in different languages to provide more resilience to the community as a whole. As of now we're pretty tied to Node.js's lifecycle and popularity.</li>
</ul>Matrix is pretty dang solid overall; there's still some room for improvement if you've got the drive and the time.Tim Cook, Facebook, and GDPR2018-10-22T13:47:00-04:002018-10-22T13:47:00-04:00https://casavant.org/technical/2018/10/22/cook-gdpr<p>Tim Cook <a href="https://www.engadget.com/2018/10/24/tim-cook-calls-for-gdpr-style-privacy-laws-in-the-us/">said this past week</a> at a privacy conference that he, and Apple, are <q>in full support of a comprehensive federal privacy law in the United States.</q> I'd like to take a look at one point in his argument.</p>
<!--more-->
<p>In support of this position, he stated:</p>
<blockquote>
<p>Platforms and algorithms that promised to improve our lives can actually magnify our worst human tendencies,<q>said Cook.</q>Rogue actors and even governments have taken advantage of user trust to deepen divisions, incite violence, and even undermine our shared sense of what is true and what is false. This crisis is real. It is not imagined, or exaggerated, or crazy.</p>
</blockquote>
<p>Specifically I am interested in the assertion that governments have done the things that he's claiming. It's obvious that they tried; we are aware of the <a href="https://en.wikipedia.org/wiki/Internet_Research_Agency">Internet Research Agency</a> in Moscow. What they did was to organize & promote rallies, create memes, and post on Twitter. None of these things would be considered unacceptable if they were done by American political groups. When these actions are considered criminal activity, we are in danger of setting up a situation in which we ask the government to step in to regulate what are clearly speech acts. Free speech is the cornerstone of liberal society; if you can't talk, you can't make change. Sometimes you won't like what's said. This is an expected part of living in such a society. The narrative in which the Internet Research Agency did something criminal is incompatible with this.</p>
<p>I also disagree with the idea that the source of an argument matters. It may help to figure out their motivations & their biases, but an argument should stand on its own. If it's unsound, this will be borne out regardless of if it was said by an American or a Russian. A set of (terrible) memes is a weak argument no matter who does the posting; were it an American who had created them we'd just consider them bad at argument (and bad at making memes), not an enemy combatant in an information war. An argument should stand on its own and these don't. And if they did, that would just show that they were right, not that they were evil.</p>
<p>There's another implication here that we citizens aren't quite smart enough not to be swayed by such lame attempts at persuasion. This is where the real risk of division comes in. It's too easy to assume that our side is enlightened and found the truth. It feels charitable to say the other side is just misled, or uninformed, or not quite as smart; we'll just have to step in and show them the way. The dirty truth is that we're all pretty dang similar. We're all humans and our intelligence levels fall along a predictable distribution, and the gap from the bottom to the top (ignoring outliers) is not so large. Let's resist the temptation to dumb down the other side. They're people. They're complex. They've developed their opinions over time the same as we have, and they've got reasons for thinking what they do.</p>
<p>We've got to treat each other as human beings. That's how you stop the division we're experiencing today.</p>
<p>Further reading:</p>
<ul>
<li><a href="https://www.reddit.com/r/ActiveMeasures/comments/74nl1w/internet_research_agency_memes/">Examples of Internet Research Agency memes, collected by a user on Reddit</a></li>
<li><a href="http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/">One of my favorite Slate Star Codex pieces on political groups</a></li>
</ul>Tim Cook said this past week at a privacy conference that he, and Apple, are in full support of a comprehensive federal privacy law in the United States. I'd like to take a look at one point in his argument.JWTs2018-10-22T13:47:00-04:002018-10-22T13:47:00-04:00https://casavant.org/technical/2018/10/22/jwt<p>There's a lot of talk about whether or not to use JSON Web Tokens and how to use them if you choose to. I'll go over some of the basics here.</p>
<!--more-->
<h2>Overview</h2>
<p>A JWT (I've always said <q>jot</q>, but I've heard <q>j-w-t</q> and <q>j-watt</q>) is a bearer token that consists of a signed set of authorization claims. Let's unpack that. Being a <q>bearer token</q> means that the token itself is considered sufficient proof of authentication. A client needs only to present the token and the server will respond. A JWT is made up of a set of claims to particular capabilities (e.g. can read email, can edit other users, can upload pictures). This set of claims is then signed by a trusted key. Because of the guarantees provided by asymmetric key cryptography, we only need the private part of the key to create the signature, not to verify it.</p>
<p>As a result we get some nice capabilities. JWTs are set up to scale. A limited number of authentication servers can issue JWTs; other servers do not need to contact the authentication servers to verify that the JWT is valid. And since the JWT contains authorization claims the other servers will also be able to provide only the capabilities that the specific user needs (or is authorized to have). All this with only one centralized user database query. This concept is called <q>stateless auth</q>.</p>
<h2>Downsides</h2>
<h3>Bearer token nature</h3>
<p>Unfortunately, since a JWT is a bearer token, it can be sniffed or phished by a third party and used to make malicious requests. To combat this, JWTs are typically issued in two tiers. A shorter-lived token that is sent with every request (the <q>access</q> token) and a longer-lived token that is only sent when requesting new access tokens (the <q>session</q> token). This way if an access token is stolen it is only valid for the rest of its lifetime (typically 15 minutes or less). If a refresh token is stolen that is a different beast, but since there's only one request every ~15 minutes, this is much rarer. And since we're all using HTTPS anyways sniffing should not be a problem (unless your system certificate trust is compromised; but then it's all over anyways, they'll just read your credentials).</p>
<h3>Logout</h3>
<p>We still have the problem of how to handle logouts. We have a few options here. When we log a user out, we've got to invalidate their tokens. Since the tokens are bearer tokens, in order to invalidate them we've got to log them in a centralized blacklist and check that when they're used. Since the refresh token is used against the authentication server anyways, we're already performing a centralized operation; it's fine to also check a blacklist. However, if we always check a blacklist when the access token is used, we lose the stateless nature of our authentication system. We have a couple options here. We could consider security paramount and use a single, centralized blacklist for all of our tokens. We could sacrifice some of our security by using a distributed eventually-consistent database as our blacklist and accept that there may be some delay before a logout event reaches all our servers. We could sacrifice a bit more of our security and not blacklist access tokens at all, tuning the access token lifetime to balance server load and logout effectiveness (notably not to the user; the user can be instructed to delete their token and they will. This issue is about malicious third parties). They've all got pros and cons. Your particular solution will depend on your security, compute, and scaling constraints.</p>
<p>As for me, I don't tend to use the two-tier architecture. I use short-lived access tokens that are able to refresh themselves. Since they can refresh themselves, I also use a blacklist. There are a couple advantages to this method. Firstly, your blacklist stays small - you can remove any tokens that have expired (and they expire in under 15 minutes). Secondly, your implementation stays simple since you only deal with a single type of token. Thirdly, user sessions automatically die if the app is closed for longer than the access token lifetime (and they don't if the app is open since the app will renew them in the background). My main constraints here are developer time and server resources, but I also don't have to scale (yet) or contain any important capabilities or information. As a result I use a simple solution that's secure enough.</p>
<h2>Other concerns</h2>
<h3>Storage</h3>
<p>JWTs are typically shown being stored into <code>localStorage</code> or <code>sessionStorage</code>. In the modern era this shouldn't be done. Both of these data stores can be read by any JavaScript that was served from the same domain. This means if you built your client-side application using third-party packages that you are implicitly trusting all the code in those packages, and all the code those packages reference, so on and so on. All of those libraries have access to those data stores. This might not be something you're worried about; we did just talk about picking a solution that's secure enough. However, you get the solution pretty much for free. Store JWTs in HTTP-only cookies. This way they're sent to your server on every request via the cookie header and are unreadable via JavaScript. Cookies are vulnerable to cross-site request forgery (enabling a third party to force you to send a particular request; they can't see your cookies or the response), but this is easy enough to combat. Protect your mutating endpoints (PUT, POST, etc.) by placing an anti-CSRF token into the JWT and into the JavaScript that the page uses. The JavaScript should then send the anti-CSRF token with future requests (the JWT in the Cookie header and the token in the X-Csrf-Token header). This will let your server validate that not only did the request come from a user with a valid JWT, but that the source of the request was the server's own JavaScript code, which is enough to prevent CSRF.</p>
<h3>Unencrypted nature</h3>
<p>JWTs are not encrypted. They look opaque, but they're just encoded signed values. It's trivial to decode them into a human-readable format (they're typically JavaScript objects, not binary values). What's not trivial is to create a signature without the server's trusted key. It's important to recognize that all a JWT guarantees us is that the value inside was not changed, not that it was not read.</p>
<h3>Old library vulnerabilities</h3>
<p>The JWT standard has a field that lets the creator specify what algorithm was used to generate the signature. It is valid to specify 'None' in this field. As a result an apparently-valid signature could be generated by anyone for any set of claims. It would be like if you went to a job interview and they first asked you what you'd like to be tested on. You'd say <q>nothing</q> and they'd respond <q>Alright then. Well done! You've got the job.</q> As a result most libraries will only consider JWTs valid if they're using a specific algorithm (since the validating server is owned by the same people who own the authentication server, this should be known in both places) which neutralizes this issue.</p>
<h2>Conclusion</h2>
<p>I think JWTs are a good idea. They make it easy to manage user sessions, are easy to implement (with a few gotchas), and scale pretty far. You should always do your research before implementing any kind of user authentication or session management, but the known issues with JWTs are pretty well known and easy to avoid.</p>
<h2>Further reading</h2>
<ul>
<li><a href="https://auth0.com/blog/critical-vulnerabilities-in-json-web-token-libraries/">On the JWT 'none' vulnerability.</a></li>
<li><a href="https://jwt.io/">A comprehensive list of JWT libraries.</a> Note the inclusion of yellow bars here with 'Minimum Version' - they reflect when the 'none' vulnerability was fixed.</li>
</ul>There's a lot of talk about whether or not to use JSON Web Tokens and how to use them if you choose to. I'll go over some of the basics here.Google envy2018-10-01T13:27:00-04:002018-10-01T13:27:00-04:00https://casavant.org/misc/2018/10/01/hiring<p>I ran across a good one this week: <a href="https://leonardofed.io/blog/startups-hiring.html">Hire people who aren't proven</a>.</p>
<!--more-->
<p>This article showed up on HN this week (<a href="https://news.ycombinator.com/item?id=18084388">comments</a>) and I think it's a great example of a general trend. The author argues that the expectations we set for potential applicants to programming jobs are artificially high:</p>
<blockquote>
<p><q>We're looking for a candidate who is: at least a one time World Cup Champion, two times FIFA World Cup awards winner, best scorer of the season, proven ability to work well in a team, exceptional references from previous teams</q>.
How many people are really a good fit for this position? Two, maybe three on Earth.</p>
</blockquote>
<p>The general trend I see is for tech companies to copy parts of the FAANG playbook. With regard to open offices in particular, I think those massive companies are succeeding <em>in spite</em> of their office design choice. Most tech workers I know don't like the open office they work in (I've met one guy who doesn't say that). The top-end set of tech companies have two things that will counteract bad cultural & environmental decisions they make: resume-building names and deep pockets. If you have an environment people don't like to work in, you can pay them more and they'll work there anyways. Barring that, if just having Google on your resume helps your career (and it does), that will push more people to work there and counteract bad decisions.</p>
<p>The economics of hiring are different in the FAANG world than they are in the startup world. The incentives affecting employees are different. Google can say <q>we need someone who lives & breathes Go and is willing to share a desk located in a storm drain</q> and they'll get ten applicants for the position; not so for anyone else. Due to the resources they already have, Google will succeed in spite of a few bad process decisions.</p>
<p>This is probably most obvious in Google's hiring process. Their interviews are built to err on the side of caution - they reject well-qualified applicants all the time. This works well for them because they have such a massive pool of applicants (and now that I say it, it wouldn't work any other way). If you've got a similar process, or even similar job descriptions, you should probably re-evaluate your hiring strategy.</p>I ran across a good one this week: Hire people who aren't proven.