duncanlock.nethttps://duncanlock.net/2024-01-25T07:13:31+00:00Sci-Fi Short Stories to Read Aloud with the Kids2024-01-25T07:13:31+00:002024-01-25T07:13:31+00:00Duncan Locktag:duncanlock.net,2024-01-25:/blog/2024/01/25/sci-fi-short-stories-to-read-aloud-with-the-kids/<p class="lead">I’ve been trying to read a Sci-Fi(ish) short story to the kids at bedtime, in between other longer books. This is how it’s been going.</p>
<section class="doc-section level-1"><h2 id="_really_successful_stories">Really Successful Stories</h2><p>These are the stories they love, that they ask for and that we’ve read multiple times.</p>
<div class="image-block"><img alt="The Mind Blown/Exploding Head Emoji" height="the top of which has exploded" src="https://duncanlock.net/images/posts/sci-fi-short-stories-to-read-aloud-with-the-kids/Noto_Emoji_v2.034_1f92f-omg.svg" width="but really huge. An image of the shocked looking cartoon face"/></div>
<p>The stories that have really <em>worked</em> generally get this kind of “mind blown” reaction from the kids at the end – which is very satisfying!</p>
<p>These are short stories by some of the <em>greatest</em> Science Fiction/<span class="caps">SF</span> authors in history. They are all well written and well put together, to say the least. These are mostly <em>not</em> written for children, but I’ve trawled the history of the genre and found some that are suitable and that my kids enjoyed.</p>
<p>I’ve found that 40s <span class="amp">&</span> 50s sci-fi shorts are sometimes quite suitable for …</p></section><p class="lead">I’ve been trying to read a Sci-Fi(ish) short story to the kids at bedtime, in between other longer books. This is how it’s been going.</p>
<section class="doc-section level-1"><h2 id="_really_successful_stories">Really Successful Stories</h2><p>These are the stories they love, that they ask for and that we’ve read multiple times.</p>
<div class="image-block"><img alt="The Mind Blown/Exploding Head Emoji" height="the top of which has exploded" src="https://duncanlock.net/images/posts/sci-fi-short-stories-to-read-aloud-with-the-kids/Noto_Emoji_v2.034_1f92f-omg.svg" width="but really huge. An image of the shocked looking cartoon face"/></div>
<p>The stories that have really <em>worked</em> generally get this kind of “mind blown” reaction from the kids at the end – which is very satisfying!</p>
<p>These are short stories by some of the <em>greatest</em> Science Fiction/<span class="caps">SF</span> authors in history. They are all well written and well put together, to say the least. These are mostly <em>not</em> written for children, but I’ve trawled the history of the genre and found some that are suitable and that my kids enjoyed.</p>
<p>I’ve found that 40s <span class="amp">&</span> 50s sci-fi shorts are sometimes quite suitable for kids. The short stories often don’t have time to be problematic (massively sexist or racist, for example) - and they’re generally pretty tame by modern standards - nobody is going to get turned inside out by a nanotech plague half way through. They are pretty hit-and-miss, though - and you <em>have to read them yourself first</em>.</p>
<p>One of the nice/annoying things about little kids, is that they have this weird memory: simultaneously an amazing, perfect recall of things they care about, and goldfish like instant wipe for stuff only you care about.</p>
<p>So, I’ve read these to them several times, months apart and get to repeat almost the same reaction each time!</p>
<section class="doc-section level-2"><h3 id="_the_egg_by_andy_weir">The Egg, by Andy Weir</h3><p>This is a <em>fabulous</em> short story, really packs a lot in and is very accessible.
It starts with a death - when I read it to the kids, I elide the details about the family <span class="amp">&</span> kids. It’s good colour, but irrelevant to the story and a bit over their heads. So I tend to read that paragraph like this:</p>
<div class="quote-block"><blockquote><p><span class="dquo">“</span>Don’t worry,” I said. “They’ll be fine. <s>Your kids will remember you as perfect in every way. They didn’t have time to grow contempt for you. Your wife will cry on the outside, but will be secretly relieved. To be fair, your marriage was falling apart. If it’s any consolation, she’ll feel very guilty for feeling relieved.</s>”</p>
<p>“Oh,” you said. “So what happens now?…</p><footer>— <cite>Andy Weir, The Egg</cite></footer></blockquote></div>
<p>This story has a great ending, that leaves the kids 🤯 every time.</p>
<p>You can read “The Egg” here: <a class="bare" href="https://galactanet.com/oneoff/theegg_mod.html">https://galactanet.com/oneoff/theegg_mod.html</a></p></section>
<section class="doc-section level-2"><h3 id="_theyre_made_of_meat_by_terry_bisson">They’re Made of Meat, by Terry Bisson</h3><p>The first time we read this - they thought it was <em>hilarious</em> - then I talked them to the realization that they’re <em>aliens</em> talking about <em>humans</em> - and they were like 🤯</p>
<p>You can read “They’re Made of Meat” here: <a class="bare" href="http://www.terrybisson.com/theyre-made-out-of-meat-2/">http://www.terrybisson.com/theyre-made-out-of-meat-2/</a></p></section>
<section class="doc-section level-2"><h3 id="_mugwump_4_by_robert_silverberg">Mugwump 4, by Robert Silverberg</h3><p>This might drag a little in the middle, but pays off at the end! This one involves time travel (which you might have to explain a bit) and has a great time-loop ending, which gets a nice 🤯 every time.</p></section>
<section class="doc-section level-2"><h3 id="_calamity_warps_by_gene_wolfe">Calamity Warps by Gene Wolfe</h3><p>This is a nice, short, fun story about a dog - who can run <em>really, really</em> fast. You can find this one in the “Starwater Strains” anthology.</p></section>
<section class="doc-section level-2"><h3 id="_dusty_zebra_by_clifford_d_simak">Dusty Zebra, by Clifford.D. Simak</h3><p>This is slightly longer, and we didn’t get to finish it in one go - but they were <em>completely</em> hooked by the story and talking the thinking about it the next day. This one doesn’t have a mind-blowing ending, but the story and ending are both quite fun.</p></section></section>
<section class="doc-section level-1"><h2 id="_somewhat_successful_ones">Somewhat Successful Ones</h2><p>These stories worked fine, but they are a bit melancholy or subdued, rather than mind-blowing or funny, which is what I’m really going for, I think. They’re good stories, and it’s nice to mix things up a little.</p>
<section class="doc-section level-2"><h3 id="_all_summer_in_a_day_by_ray_bradbury">All Summer in a Day, by Ray Bradbury</h3><p>This is very accessible - it’s about kids at school - but set on Venus. It’s not <em>actual</em> Venus, it’s a Ray Bradbury imagined Venus from the 50s before we’d sent any probes or landers to actual Venus. So it’s habitable, just very cloudy and rains all the time.
The setting is interesting, kids behave badly and there are melancholy consequences.</p></section>
<section class="doc-section level-2"><h3 id="_bears_discover_fire_by_terry_bisson">Bears Discover Fire, by Terry Bisson</h3><p>I really like this one - it’s quite atmospheric and rather “Americana”, which is difficult for me to convey fully when I read it, I think (given that we don’t live in America and my very not American accent). The story is interesting, but surprisingly mellow <em>and</em> melancholy, given the title. There is a death - an old Grandma passes away in her sleep.</p>
<p>The six-year-old fell asleep, the 9 yr old liked it.</p></section></section>
<section class="doc-section level-1"><h2 id="_less_successful_ones">Less Successful Ones</h2><p>These stories are great, but didn’t work quite so well with my kids.</p>
<section class="doc-section level-2"><h3 id="_when_the_yogurt_took_over_by_john_scalzi">When the Yogurt Took Over, by John Scalzi</h3><p>This is a <em>great</em> little short story, but hasn’t worked all that well for my kids. It <em>nearly</em> works, but it’s a bit too dry and sardonic and pitched at adults a bit too much. They say they want to hear it, because the title sounds hilarious, but they’re a bit nonplussed by the actual story.</p>
<p>The six-year-old fell asleep, the 9 yr old didn’t.</p>
<p>Read “When the Yogurt Took Over” here: <a class="bare" href="https://whatever.scalzi.com/2010/10/02/when-the-yogurt-took-over-a-short-story/">https://whatever.scalzi.com/2010/10/02/when-the-yogurt-took-over-a-short-story/</a></p></section>
<section class="doc-section level-2"><h3 id="_the_nine_billion_names_of_god_arthur_c_clarke">The Nine Billion Names of God, Arthur C. Clarke</h3><p>I love this, but it didn’t make much impact on the kids. There’s a bit too much 50s <span class="caps">IBM</span> stuff and not enough action - it’s quite restrained. It has a mind-blowing ending that works on adults but didn’t work for my kids, when I read it. Might try again when they’re older.</p></section></section>
<section class="doc-section level-1"><h2 id="_any_suggestions">Any Suggestions?</h2><p>I’m looking for suggestions for more! What stories would you recommend? <a href="https://cosocial.ca/@duncanlock/111845295291721512">Let me know</a>!</p></section>Super Fast Reader Mode for the Entire Web, with Dillo Plus2024-01-04T00:32:59+00:002024-01-15T11:15:21+00:00Duncan Locktag:duncanlock.net,2024-01-04:/blog/2024/01/04/super-fast-reader-mode-for-the-entire-web-with-dillo-plus/<p class="lead">Want “Reader Mode” for almost the entire web - and have every page load nearly instantly? Here’s how.</p>
<p>There are three main constellations of Web Browsers - ones based on Google’s Blink <a href="https://en.wikipedia.org/wiki/Comparison_of_browser_engines">rendering engine</a> (used in Google Chrome, <span class="caps">MS</span> Edge, Vivaldi, Brave and others), ones based on WebKit (used in Safari on MacOS <span class="amp">&</span> iOS 7 iPadOS) - and on Mozilla’s Geko engine (used by <a href="https://www.mozilla.org/firefox/">Firefox</a> <span class="amp">&</span> Thunderbird).</p>
<p>There are also a few other, <em>“non-mainstream browsers”</em>, that have their <em>own</em> engines. Cosmic wanderers, not part of the main constellations, that chart their own course, building everything themselves: <a href="https://en.wikipedia.org/wiki/Flow_(web_browser)">Flow</a>, <a href="https://servo.org/">Servo</a>, <a href="https://ladybird.dev/">LadyBird</a>, <a href="https://www.netsurf-browser.org/">NetSurf</a> - and <a href="https://github.com/crossbowerbt/dillo-plus">Dillo-Plus</a>.</p>
<section class="doc-section level-1"><h2 id="_dillo_plus_has_a_tiny_superpower">Dillo Plus has a Tiny Superpower</h2><p>Actually, the Dillo-Plus browser has quite a few <a href="https://github.com/crossbowerbt/dillo-plus#browser-features">tiny superpowers</a> - but the one I’m interested in is its ability to <em>ignore all the <span class="caps">CSS</span> styles from the website and load your <span class="caps">CSS …</span></em></p></section><p class="lead">Want “Reader Mode” for almost the entire web - and have every page load nearly instantly? Here’s how.</p>
<p>There are three main constellations of Web Browsers - ones based on Google’s Blink <a href="https://en.wikipedia.org/wiki/Comparison_of_browser_engines">rendering engine</a> (used in Google Chrome, <span class="caps">MS</span> Edge, Vivaldi, Brave and others), ones based on WebKit (used in Safari on MacOS <span class="amp">&</span> iOS 7 iPadOS) - and on Mozilla’s Geko engine (used by <a href="https://www.mozilla.org/firefox/">Firefox</a> <span class="amp">&</span> Thunderbird).</p>
<p>There are also a few other, <em>“non-mainstream browsers”</em>, that have their <em>own</em> engines. Cosmic wanderers, not part of the main constellations, that chart their own course, building everything themselves: <a href="https://en.wikipedia.org/wiki/Flow_(web_browser)">Flow</a>, <a href="https://servo.org/">Servo</a>, <a href="https://ladybird.dev/">LadyBird</a>, <a href="https://www.netsurf-browser.org/">NetSurf</a> - and <a href="https://github.com/crossbowerbt/dillo-plus">Dillo-Plus</a>.</p>
<section class="doc-section level-1"><h2 id="_dillo_plus_has_a_tiny_superpower">Dillo Plus has a Tiny Superpower</h2><p>Actually, the Dillo-Plus browser has quite a few <a href="https://github.com/crossbowerbt/dillo-plus#browser-features">tiny superpowers</a> - but the one I’m interested in is its ability to <em>ignore all the <span class="caps">CSS</span> styles from the website and load your <span class="caps">CSS</span> instead</em>:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/super-fast-reader-mode-for-the-entire-web-with-dillo-plus/before-and-after-3.png" alt="Screenshots of three web pages, each with two browser windows side by side - one of Firefox showing the normal appearance and one with Dillo Plus and my stylesheet, showing the Reader Mode version.">
<figcaption>Figure 1. Before and After - Firefox on the left, Dillo Plus (and my stylesheet) on the right.</figcaption></figure>
<p>Most browsers except Chrome support “User <span class="caps">CSS</span>” in some form, although they mostly don’t advertise this and make it difficult to set up. But, as far as I know, they all load the website’s <span class="caps">CSS</span> styles <em>first</em>, then apply your user styles <em>on top</em>.</p>
<p>This is fine – and is how the “Cascading” in Cascading Style Sheets is <em>supposed</em> to work, after all. It allows you to override things you don’t like, and this works <span class="caps">OK</span>-ish for little things – but you’re always <em>fighting</em> with the styles from the website.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/super-fast-reader-mode-for-the-entire-web-with-dillo-plus/dillo-tools-menu.png" alt="Screenshot of the Dillo Plus tools menu, showing the options: 'Use remote CSS' (unchecked), 'Use embedded CSS' (unchecked) - and 'Use reader mode CSS', which is checked.">
<figcaption>Figure 2. Use the “Tools” menu to switch off all other <span class="caps">CSS</span> (uncheck the first two options) - and switch on your Reader/User Mode <span class="caps">CSS</span>.</figcaption></figure>
<p>If you want to override <em>lots of things</em> (or everything), you tend to end up with large, unwieldy <span class="caps">CSS</span> that overrides tons of things, as well as lots of little <span class="caps">CSS</span> overrides for individual websites - and it ends up full of <code>!important</code> qualifiers to make your styles override previously applied ones.</p>
<p>Dillo Plus allows you to avoid all that messing around and just <em><strong>switch off all other <span class="caps">CSS</span> except yours</strong></em>. This sounds pretty basic, but I don’t know of any other browser that supports this - and the difference it makes is <em>massive</em>.</p></section>
<section class="doc-section level-1"><h2 id="_your_web_your_way">Your Web, Your Way</h2><p>Once you do this, <em>all</em> the websites you visit will suddenly look the way <em>you</em> want them to. Want everything in Dark mode? Want everything to use the same font? Want all the fonts to be larger, everywhere? No problem, done. Whatever you want, everywhere, all at once.</p>
<p><em><strong>Once you browse around this way for a while, the web starts to feel a little bit like one giant website, that’s so fast it’s just…​ part of your computer.</strong></em></p></section>
<section class="doc-section level-1"><h2 id="_bring_that_smolnetgemini_simplicity_back_to_the_web">Bring that Smolnet/Gemini Simplicity (back) to the Web</h2><p>If you’ve used <a href="https://en.wikipedia.org/wiki/Gemini_(protocol)">Gemini</a>, or <a href="https://en.wikipedia.org/wiki/Gopher_(protocol)">Gopher</a>, or the early Web, you will know what I mean here, but it’s a feeling of simplicity and coherence, as well as a more homespun feel, rather than flashy commercial design in your face all the time. It’s…​ nice, if you like that sort of thing.</p>
<p>As a bonus, Dillo-Plus <em>not only</em> supports Gemini <span class="amp">&</span> Gopher, but <em>also</em> applies all this <span class="caps">CSS</span> magic to Gemini <span class="amp">&</span> Gopher <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a> pages, making the whole web smol!</p></section>
<section class="doc-section level-1"><h2 id="_get_dillo_plus">Get Dillo Plus</h2><p>You can get <a href="https://github.com/crossbowerbt/dillo-plus">Dillo-Plus here</a>, but you are going to have to build it from source yourself. Yes, I <em>know</em>. Doing this on Linux is pretty painless, only takes a minute and just worked for me. I have not tried any other platforms, although it supports lots. This is how I did it on Ubuntu 22.04:</p>
<p>First check out the source code:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>git clone git@github.com:crossbowerbt/dillo-plus.git
<span class="gp">$</span><span class="w"> </span><span class="nb">cd </span>dillo-plus</code></pre></div>
<p>Check if you have the required build dependencies installed:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>fltk-config <span class="nt">--version</span>
<span class="go">// I didn't, so I installed it:
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt <span class="nb">install </span>libfltk1.3-dev</code></pre></div>
<p>Build the browser:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>make clean <span class="o">&&</span> make</code></pre></div>
<p>Install it - this will overwrite any existing Dillo you might have installed:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>make <span class="nb">install</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_the_reader_mode_user_stylesheet">The Reader Mode User Stylesheet</h2><p>Dillo-Plus does ship with a built-in Reader Mode Stylesheet, but I made one that I like much better. Bear in mind that Dillo doesn’t support fancy new <span class="caps">CSS</span> things like variables, or units like <code>rem</code>, so I’ve avoided those:</p>
<div class="listing-block scrollable"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* dark mode, and a little bit of meyer reset */</span>
<span class="o">*</span> <span class="p">{</span>
<span class="nl">background-color</span><span class="p">:</span> <span class="m">#363839</span> <span class="cp">!important</span><span class="p">;</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#d1cec9</span> <span class="cp">!important</span><span class="p">;</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">100%</span><span class="p">;</span>
<span class="nl">font</span><span class="p">:</span> <span class="nb">inherit</span><span class="p">;</span>
<span class="nl">vertical-align</span><span class="p">:</span> <span class="nb">baseline</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">body</span> <span class="p">{</span>
<span class="nl">line-height</span><span class="p">:</span> <span class="m">1.1</span><span class="p">;</span>
<span class="nl">max-width</span><span class="p">:</span> <span class="m">85%</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">3em</span><span class="p">;</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">150%</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">h1</span><span class="o">,</span>
<span class="nt">h2</span><span class="o">,</span>
<span class="nt">h3</span><span class="o">,</span>
<span class="nt">h4</span><span class="o">,</span>
<span class="nt">h5</span><span class="o">,</span>
<span class="nt">h6</span><span class="o">,</span>
<span class="nt">p</span> <span class="p">{</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#d1cec9</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">p</span> <span class="p">{</span>
<span class="nl">line-height</span><span class="p">:</span> <span class="m">1.35</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">/* https://alistapart.com/article/axiomatic-css-and-lobotomized-owls/ */</span>
<span class="o">*</span> <span class="o">+</span> <span class="o">*</span> <span class="p">{</span>
<span class="nl">margin-top</span><span class="p">:</span> <span class="m">1.2em</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">/* Heading Hierarchy */</span>
<span class="nt">h1</span> <span class="p">{</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">60px</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">h2</span> <span class="p">{</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">48px</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">h3</span> <span class="p">{</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">36px</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">h4</span> <span class="p">{</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">30px</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">h5</span> <span class="p">{</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">24px</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">h6</span> <span class="p">{</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">21px</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">/* Links */</span>
<span class="nt">a</span> <span class="p">{</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#d1cec9</span><span class="p">;</span>
<span class="nl">text-decoration</span><span class="p">:</span> <span class="nb">underline</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">a</span><span class="nd">:visited</span> <span class="p">{</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#8e8b88</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">blockquote</span><span class="o">,</span>
<span class="nt">aside</span> <span class="p">{</span>
<span class="nl">border-left</span><span class="p">:</span> <span class="m">5px</span> <span class="nb">solid</span> <span class="no">gray</span><span class="p">;</span>
<span class="nl">margin-left</span><span class="p">:</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">padding-left</span><span class="p">:</span> <span class="m">1em</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">/* Code */</span>
<span class="nt">code</span><span class="o">,</span>
<span class="nt">pre</span> <span class="p">{</span>
<span class="nl">font-family</span><span class="p">:</span> <span class="nb">monospace</span><span class="p">;</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">95%</span><span class="p">;</span>
<span class="nl">border</span><span class="p">:</span> <span class="m">1px</span> <span class="nb">dotted</span> <span class="m">#ccc</span><span class="p">;</span>
<span class="nl">padding</span><span class="p">:</span> <span class="m">0.4em</span> <span class="m">0.5em</span><span class="p">;</span>
<span class="c">/* wrap */</span>
<span class="nl">white-space</span><span class="p">:</span> <span class="n">pre-wrap</span><span class="p">;</span>
<span class="nl">word-wrap</span><span class="p">:</span> <span class="n">break-word</span><span class="p">;</span>
<span class="p">}</span></code></pre></div>
<p>That’s it, currently - 84 lines of <span class="caps">CSS</span>, including blank lines <span class="amp">&</span> comments, to restyle the entire web!</p>
<p>I’ve put this <a href="https://github.com/dflock/dillo-reader-mode-css">up on GitHub here</a> if you want to grab it that way, contribute updates or add new ones!</p>
<section class="doc-section level-2"><h3 id="_make_dillo_plus_use_your_stylesheet">Make Dillo Plus Use Your Stylesheet</h3><p>Dillo Plus puts its reader mode stylesheet here: <code>/usr/local/etc/dillo/style_reader_mode.css</code>, which is owned by <code>root</code>. So you will need to either edit this as <code>root</code>, or to make things easier, replace the bundled user stylesheet with symlink to your one, so you can edit it without being <code>root</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo rm</span> /usr/local/etc/dillo/style_reader_mode.css
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo ln</span> <span class="nt">-s</span> ~/dev/dillo-reader-mode-css/simple-dark.css /usr/local/etc/dillo/style_reader_mode.css</code></pre></div>
<aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>Once you’ve done this, dillo will load your stylesheet on every page refresh, so you can just edit, save and refresh to see your changes!</p></aside></section></section>
<section class="doc-section level-1"><h2 id="_upsides_downsides_of_browsing_like_its_1999">Upsides <span class="amp">&</span> Downsides of Browsing like it’s 1999</h2><p>So, this isn’t all gravy - there <em>are</em> some downsides to browsing like it’s 1999:</p>
<section class="doc-section level-2"><h3 id="_upsides">Upsides</h3><div class="ulist"><ul><li>No JavaScript, which basically means no ads or malware either.</li><li>Dillo is fast anyway, but no JavaScript <em>really</em> helps with this.</li><li>Like, really fast.</li><li>Your web, your way. Dark Mode? Done. Larger fonts? Done. Rainbow Background? Done. Whatever you want, everywhere, all at once.</li><li>Bring that smolnet/Gemini simplicity (back) to the web.</li></ul></div></section>
<section class="doc-section level-2"><h3 id="_downsides">Downsides</h3><figure class="image-block"><img src="https://duncanlock.net/images/posts/super-fast-reader-mode-for-the-entire-web-with-dillo-plus/new-york-times.png" alt="Screenshot of a New York Times article, Firefox on the left, Dillo Plus on the right. In the Firefox window, the article is obscured by Cookie popups and Paywalls. The Dillo Plus one is blank except for 'Please enable JS and disable any ad blocker'.">
<figcaption>Figure 3. The New York Times - Firefox on the left, Dillo Plus on the right. You can’t read the article either way, but still…​</figcaption></figure>
<div class="ulist"><ul><li>No JavaScript, so websites that rely on JavaScript just don’t work.<ul><li>Medium Articles load images via JavaScript with (apparently) no fallback, for example, so no images in Medium.</li></ul></li><li>Currently no <span class="caps">SVG</span> or Webp, <span class="caps">AVIF</span> etc…​ image support.</li><li>You kind of get a taste of the “Screen Reader” experience of the Web - this is mostly that people building the websites aren’t given time to build them with your browser in mind, nor testing in your setup, so things break/suck/work ok, depending.</li><li>Commercial websites tend to have a <em>lot</em> of navigation crap at the top - loads of nested lists of links - usually turned into dropdown menu’s with <span class="caps">CSS</span> <span class="amp">&</span> <span class="caps">JS</span>. This will all get spat out at the top of the page. Some sites have a “Skip to Content” link at the top, which is <em>very</em> useful - but many don’t.</li><li>You have to build the browser from source code yourself. If you’re on Linux, don’t worry, it builds in a few seconds and just works.</li></ul></div></section></section>
<section class="doc-section level-1"><h2 id="_could_firefox_support_this_please">Could Firefox Support This, Please?</h2><p>It would be really nice to have the option to do this in Firefox. That way you’d get all the support for modern <span class="caps">CSS</span> <span class="amp">&</span> image formats, as well as the ability to switch off all <span class="caps">CSS</span> except your own. These browsers <em>can</em> already switch off JavaScript, although the option is rather buried in their settings, but not <span class="caps">CSS</span>.</p>
<hr></section>
<section class="doc-section level-1"><h2 id="_discussion">Discussion</h2><div class="olist arabic"><ol class="arabic"><li><a href="https://cosocial.ca/@duncanlock/111760797567309929">Discuss on Mastodon</a></li><li><a href="https://news.ycombinator.com/item?id=38995577">Discuss on Hacker News</a></li></ol></div>
<section class="doc-section level-2"><h3 id="_footnotes">Footnotes</h3></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">Gemini pages look like web pages, Gopher pages look like Gopher pages - i.e. preformatted text <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>Thanks, David Peter (sharkdp)!2023-05-28T10:15:05-07:002023-06-03T09:23:36-07:00Duncan Locktag:duncanlock.net,2023-05-28:/blog/2023/05/28/thanks-david-peter-sharkdp/<p>It’s too easy to overlook the countless hours of dedication poured into the open source software that powers our digital lives. I want to take a moment to express my appreciation and admiration for one of the many unsung heroes of my digital world, who’s software I use all the time.</p>
<section class="doc-section level-1"><h2 id="_david_peter_sharkdp">David Peter / sharkdp</h2><section class="doc-section level-2"><h3 id="_github_profile"><a href="https://github.com/sharkdp">GitHub Profile</a></h3><figure class="image-block"><a class="image" href="https://github.com/sharkdp"><img src="https://avatars.githubusercontent.com/u/4209276?v=4" alt="Avatar" width="100"></a>
<figcaption><a href="https://github.com/sharkdp">GitHub Profile</a></figcaption></figure>
<div class="ulist"><ul><li>Location: Stuttgart, Germany</li><li>Followers: 5684</li><li><a href="https://github.com/sharkdp?tab=repositories&type=source">Public Repositories</a>: 104</li><li><a class="bare" href="https://david-peter.de/">https://david-peter.de/</a></li><li><img src="https://duncanlock.net/images/icons/custom/github_sponsor_heart.svg" alt="github sponsor heart" class="icon"> <a href="https://github.com/sponsors/sharkdp">Sponsor David Peter (sharkdp)</a></li></ul></div>
<p class="lead clear">These are the projects of theirs that I love <span class="amp">&</span> use the most:</p></section>
<section class="doc-section level-2"><h3 id="_bat"><a href="https://github.com/sharkdp/bat">bat</a></h3><div class="quote-block"><blockquote><p>A cat(1) clone with wings.</p></blockquote></div>
<p>Bat is a <code>cat</code> clone with syntax highlighting for loads of languages, Git integration - and it’s written in Rust, so it works on Linux/Windows/MacOS. It’s completely replaced <code>cat</code> for me:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="shell"><span class="nb">alias cat</span><span class="o">=</span><span class="s2">"bat"</span></code></pre></div></section>
<section class="doc-section level-2"><h3 id="_fd"><a href="https://github.com/sharkdp/fd">fd</a></h3><div class="quote-block"><blockquote><p>A simple, fast and user-friendly alternative to ‘find …</p></blockquote></div></section></section><p>It’s too easy to overlook the countless hours of dedication poured into the open source software that powers our digital lives. I want to take a moment to express my appreciation and admiration for one of the many unsung heroes of my digital world, who’s software I use all the time.</p>
<section class="doc-section level-1"><h2 id="_david_peter_sharkdp">David Peter / sharkdp</h2><section class="doc-section level-2"><h3 id="_github_profile"><a href="https://github.com/sharkdp">GitHub Profile</a></h3><figure class="image-block"><a class="image" href="https://github.com/sharkdp"><img src="https://avatars.githubusercontent.com/u/4209276?v=4" alt="Avatar" width="100"></a>
<figcaption><a href="https://github.com/sharkdp">GitHub Profile</a></figcaption></figure>
<div class="ulist"><ul><li>Location: Stuttgart, Germany</li><li>Followers: 5684</li><li><a href="https://github.com/sharkdp?tab=repositories&type=source">Public Repositories</a>: 104</li><li><a class="bare" href="https://david-peter.de/">https://david-peter.de/</a></li><li><img src="https://duncanlock.net/images/icons/custom/github_sponsor_heart.svg" alt="github sponsor heart" class="icon"> <a href="https://github.com/sponsors/sharkdp">Sponsor David Peter (sharkdp)</a></li></ul></div>
<p class="lead clear">These are the projects of theirs that I love <span class="amp">&</span> use the most:</p></section>
<section class="doc-section level-2"><h3 id="_bat"><a href="https://github.com/sharkdp/bat">bat</a></h3><div class="quote-block"><blockquote><p>A cat(1) clone with wings.</p></blockquote></div>
<p>Bat is a <code>cat</code> clone with syntax highlighting for loads of languages, Git integration - and it’s written in Rust, so it works on Linux/Windows/MacOS. It’s completely replaced <code>cat</code> for me:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="shell"><span class="nb">alias cat</span><span class="o">=</span><span class="s2">"bat"</span></code></pre></div></section>
<section class="doc-section level-2"><h3 id="_fd"><a href="https://github.com/sharkdp/fd">fd</a></h3><div class="quote-block"><blockquote><p>A simple, fast and user-friendly alternative to ‘find’</p></blockquote></div>
<p>fd is very fast (parallelized directory traversal), flexible, and intuitive (<code>fd PATTERN</code> instead of <code>find -iname '<strong>PATTERN</strong>'</code>) utility for searching files and folders. It supports regular expressions or glob-based patterns, color-coded file type highlighting, and smart case-insensitivity. It ignores hidden files and directories and patterns from .gitignore by default - and also supports parallel command execution.</p>
<p>This is what <code>tldr fd</code> says:</p>
<div class="dlist"><dl><dt>Recursively find files matching a specific pattern in the current directory</dt><dd><code>fd "string|regex"</code></dd><dt>Find files that begin with <code>foo</code></dt><dd><code>fd "^foo"</code></dd><dt>Find files with a specific extension</dt><dd><code>fd --extension txt</code></dd><dt>Find files in a specific directory</dt><dd><code>fd "string|regex" path/to/directory</code></dd><dt>Include ignored and hidden files in the search</dt><dd><code>fd --hidden --no-ignore "string|regex"</code></dd><dt>Run a command on each search result returned</dt><dd><code>fd "string|regex" --exec command`</code></dd></dl></div></section>
<section class="doc-section level-2"><h3 id="_hyperfine"><a href="https://github.com/sharkdp/hyperfine">hyperfine</a></h3><div class="quote-block"><blockquote><p>A command-line benchmarking tool</p></blockquote></div>
<p>I don’t always run benchmarks, but when I do I use Hyperfine! This is much better than running the traditional <code>time <cmd></code>:</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/thanks-david-peter/hyperfine.gif" alt="Short video of Hyperfine working in the console."></div>
<p>This is what <code>tldr fd</code> says:</p>
<div class="dlist"><dl><dt>Run a basic benchmark, performing at least 10 runs</dt><dd><code>hyperfine 'make'</code></dd><dt>Run a comparative benchmark</dt><dd><code>hyperfine 'make target1' 'make target2'</code></dd><dt>Change minimum number of benchmarking runs</dt><dd><code>hyperfine --min-runs 7 'make'</code></dd><dt>Perform benchmark with warmup</dt><dd><code>hyperfine --warmup 5 'make'</code></dd><dt>Run a command before each benchmark run (to clear caches, etc.)</dt><dd><code>hyperfine --prepare 'make clean' 'make'</code></dd><dt>Run a benchmark where a single parameter changes for each run</dt><dd><code>hyperfine --prepare 'make clean' --parameter-scan num_threads 1 10 'make -j {num_threads}'</code></dd></dl></div>
<hr></section>
<section class="doc-section level-2"><h3 id="_sponsor_david_peter_sharkdp">Sponsor David Peter (sharkdp)</h3><div class="image-block"><img src="https://duncanlock.net/images/icons/custom/github_sponsor_heart.svg" alt="100" width="100"></div>
<p class="lead">If you use any of their software, or just want to support the magnificent work of David Peter (sharkdp) – you should <a href="https://github.com/sponsors/sharkdp">sponsor them on GitHub.</a></p></section></section>Debugging with an existing browser instance, or Brave in VSCode2023-05-27T09:51:30-07:002023-05-27T09:51:30-07:00Duncan Locktag:duncanlock.net,2023-05-27:/blog/2023/05/27/debugging-with-an-existing-browser-instance-or-brave-in-vscode/<p>If you want to use an existing instance of a browser to debug in VSCode - instead of always launching a new one - you have to:</p>
<div class="ulist"><ul><li>Change the shortcut that you use to start the browser, to add <code>--remote-debugging-port=9222</code> on the end of the command line, so that it always runs in “debug ready” mode.</li><li>Use a wildcard for the <code>url</code> in your <code>launch.js</code> config file.</li></ul></div>
<p>To debug web apps in VSCode with <a href="https://brave.com/">Brave</a> - add this to your <code>launch.js</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="json"><span class="p">{</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.2.0"</span><span class="p">,</span><span class="w">
</span><span class="nl">"configurations"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Launch Brave against localhost"</span><span class="p">,</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"chrome"</span><span class="p">,</span><span class="w">
</span><span class="nl">"request"</span><span class="p">:</span><span class="w"> </span><span class="s2">"launch"</span><span class="p">,</span><span class="w">
</span><span class="nl">"port"</span><span class="p">:</span><span class="w"> </span><span class="mi">9222</span><span class="p">,</span><span class="w">
</span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://localhost:3333/*"</span><span class="p">,</span><span class="w">
</span><span class="nl">"userDataDir"</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="nl">"webRoot"</span><span class="p">:</span><span class="w"> </span><span class="s2">"${workspaceFolder}"</span><span class="p">,</span><span class="w">
</span><span class="nl">"runtimeExecutable"</span><span class="p">:</span><span class="w"> </span><span class="s2">"C:/Users/DuncanLock/AppData/Local/BraveSoftware/Brave-Browser/Application/brave.exe"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span></code></pre></div>Thanks, Zachary Yedidia (zyedidia)!2023-05-25T10:15:06-07:002023-06-03T10:19:30-07:00Duncan Locktag:duncanlock.net,2023-05-25:/blog/2023/05/25/thanks-zachary-yedidia-zyedidia/<p>Open source developers are often the unsung heroes of the technology world, creating <span class="amp">&</span> maintaining the software that powers our digital lives. Sadly, their contributions often go unnoticed, but without their dedication and expertise, much of the software <span class="amp">&</span> digital infrastructure what we take for granted today wouldn’t exist.</p>
<p>I want to recognize the brilliant work of one developer and express my gratitude to them for making the world a better place through open source.</p>
<section class="doc-section level-1"><h2 id="_zachary_yedidia_zyedidia">Zachary Yedidia / zyedidia</h2><section class="doc-section level-2"><h3 id="_github_profile"><a href="https://github.com/zyedidia">GitHub Profile</a></h3><figure class="image-block"><a class="image" href="https://github.com/zyedidia"><img src="https://avatars.githubusercontent.com/u/5513065?v=4" alt="Avatar" width="100"></a>
<figcaption><a href="https://github.com/zyedidia">GitHub Profile</a></figcaption></figure>
<div class="ulist"><ul><li>Location: Stanford, <span class="caps">CA</span></li><li>Followers: 644</li><li><a href="https://github.com/zyedidia?tab=repositories&type=source">Public Repositories</a>: 175</li><li>Stanford University</li><li><a class="bare" href="https://zyedidia.github.io">https://zyedidia.github.io</a></li></ul></div>
<p class="lead clear">These are the projects of theirs that I love <span class="amp">&</span> use the most:</p></section>
<section class="doc-section level-2"><h3 id="_eget"><a href="https://github.com/zyedidia/eget">eget</a></h3><div class="quote-block"><blockquote><p>Easily install prebuilt binaries from GitHub.</p></blockquote></div>
<p>Most of the software on my machine comes from the systems package manager, and is automatically kept up to date. The problems with this, is that …</p></section></section><p>Open source developers are often the unsung heroes of the technology world, creating <span class="amp">&</span> maintaining the software that powers our digital lives. Sadly, their contributions often go unnoticed, but without their dedication and expertise, much of the software <span class="amp">&</span> digital infrastructure what we take for granted today wouldn’t exist.</p>
<p>I want to recognize the brilliant work of one developer and express my gratitude to them for making the world a better place through open source.</p>
<section class="doc-section level-1"><h2 id="_zachary_yedidia_zyedidia">Zachary Yedidia / zyedidia</h2><section class="doc-section level-2"><h3 id="_github_profile"><a href="https://github.com/zyedidia">GitHub Profile</a></h3><figure class="image-block"><a class="image" href="https://github.com/zyedidia"><img src="https://avatars.githubusercontent.com/u/5513065?v=4" alt="Avatar" width="100"></a>
<figcaption><a href="https://github.com/zyedidia">GitHub Profile</a></figcaption></figure>
<div class="ulist"><ul><li>Location: Stanford, <span class="caps">CA</span></li><li>Followers: 644</li><li><a href="https://github.com/zyedidia?tab=repositories&type=source">Public Repositories</a>: 175</li><li>Stanford University</li><li><a class="bare" href="https://zyedidia.github.io">https://zyedidia.github.io</a></li></ul></div>
<p class="lead clear">These are the projects of theirs that I love <span class="amp">&</span> use the most:</p></section>
<section class="doc-section level-2"><h3 id="_eget"><a href="https://github.com/zyedidia/eget">eget</a></h3><div class="quote-block"><blockquote><p>Easily install prebuilt binaries from GitHub.</p></blockquote></div>
<p>Most of the software on my machine comes from the systems package manager, and is automatically kept up to date. The problems with this, is that packages in the system repositories are deliberately stable - and not kept up to date with the latest changes; system repos also don’t contain newly released software.</p>
<p>That leaves a small collection of software that is either too new - or I want a newer version of - that I need to install <span class="amp">&</span> update manually.</p>
<p>That stuff is often on GitHub - and if you want it, you need to download it and put it in your <code>~/bin</code> folder yourself - and keep it updated yourself too.</p>
<p>Eget is a great, simple way to automate that.</p>
<p>I have a little <code>~/bin/update-eget.sh</code> script, that’s included in my larger <code>~/bin/update-system.sh</code>, that updates all those things to their latest available version:</p>
<div class="listing-block scrollable"><pre class="rouge highlight"><code data-lang="shell"><span class="c">#!/usr/bin/env bash</span>
<span class="nb">cd</span> ~/bin <span class="o">||</span> <span class="nb">exit</span>
<span class="c"># Token is in ~/.config/eget/eget.toml</span>
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating eget..."</span>
eget zyedidia/eget
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating micro..."</span>
eget zyedidia/micro <span class="nt">--asset</span> static
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating pandoc..."</span>
eget jgm/pandoc
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating fzf..."</span>
eget junegunn/fzf
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating eza..."</span>
eget eza-community/eza <span class="nt">--asset</span> musl.tar.gz
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating starship..."</span>
eget starship/starship <span class="nt">--asset</span> ^musl
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating bat..."</span>
eget sharkdp/bat <span class="nt">--asset</span> ^musl
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating btop..."</span>
eget aristocratos/btop
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating duf..."</span>
eget muesli/duf <span class="nt">--asset</span> x86_64.tar.gz
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating caddy..."</span>
eget caddyserver/caddy <span class="nt">--asset</span> amd64.tar.gz <span class="nt">--asset</span> ^.sig
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating vale..."</span>
eget errata-ai/vale <span class="nt">--asset</span> 64-bit.tar
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating lapce..."</span>
eget lapce/lapce <span class="nt">--asset</span> Lapce-linux.tar.gz
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating pdftilecut..."</span>
eget oxplot/pdftilecut
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating tealdeer..."</span>
eget dbrgn/tealdeer
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating fd..."</span>
eget sharkdp/fd <span class="nt">--asset</span> x86_64-unknown-linux-musl.tar.gz
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating pdfcpu..."</span>
eget pdfcpu/pdfcpu
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating sd..."</span>
eget chmln/sd <span class="nt">--asset</span> ^gnu
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">These ones always update, regardless...</span><span class="se">\n</span><span class="s2">"</span>
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating tlrc..."</span>
eget tldr-pages/tlrc <span class="nt">--asset</span><span class="o">=</span>linux-gnu.tar.gz <span class="nt">--file</span><span class="o">=</span>tldr
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating cascadia-code font..."</span>
eget microsoft/cascadia-code <span class="nt">--file</span><span class="o">=</span>ttf/<span class="k">*</span>.ttf <span class="nt">--all</span> <span class="nt">--to</span> ~/.fonts
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating rg..."</span>
eget BurntSushi/ripgrep
<span class="nb">echo</span> <span class="nt">-e</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">Updating rclone..."</span><span class="nb">sudo</span> /home/duncan/bin/eget rclone/rclone <span class="nt">--asset</span> amd64.zip <span class="nt">--to</span> /usr/bin</code></pre></div>
<p>Here’s what my eget config file (<code>~/.config/eget/eget.toml</code>) looks like:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="toml"><span class="nn">[global]</span>
<span class="py">target</span> <span class="p">=</span> <span class="s">"~/bin"</span>
<span class="py">upgrade_only</span> <span class="p">=</span> <span class="kc">true</span>
<span class="py">github_token</span> <span class="p">=</span> <span class="s">"ghp_..."</span></code></pre></div></section>
<section class="doc-section level-2"><h3 id="_micro"><a href="https://github.com/zyedidia/micro">micro</a></h3><div class="quote-block"><blockquote><p>A modern and intuitive terminal-based text editor</p></blockquote></div>
<p>Micro is my <a href="https://duncanlock.net/blog/2021/11/13/three-editor-use-cases/">command line/terminal editor of choice</a>.
I can’t say enough good things about it - it’s simple, fast, has good syntax highlighting, line numbers, Unicode support, soft wrapping - and supports all the normal editing hotkeys out of the box: Ctrl+x,c,v for Cut, Copy <span class="amp">&</span> Paste, Ctrl+s for Save, Ctrl+z undo!</p>
<p>Put this in your <code>.bashrc</code> and never look back:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="shell"><span class="nb">export </span><span class="nv">EDITOR</span><span class="o">=</span>micro
<span class="nb">export </span><span class="nv">MICRO_TRUECOLOR</span><span class="o">=</span>1</code></pre></div>
<p>It’s written in Go and comes as a single static binary that you can just download to anywhere and just run. It’s completely replaced nano <span class="amp">&</span> vi and is the only cli editor that I use. If you’re editing in a terminal, you should use micro.</p></section></section>Using git hashes in Vite & VueJS2023-05-23T10:22:36-07:002023-05-23T10:22:36-07:00Duncan Locktag:duncanlock.net,2023-05-23:/blog/2023/05/23/using-git-hashes-in-vite-vuejs/<p>To use a git hash in Vite, install <a href="https://github.com/kurttheviking/git-rev-sync-js">git-rev-sync</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="consle">$ pnpm install git-rev-sync --save</code></pre></div>
<p>then add this to your <code>vite.config.ts</code> file:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="ts"><span class="c1">// Make some git info available as env var - must start with VITE_</span>
<span class="k">import</span> <span class="nx">git</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">git-rev-sync</span><span class="dl">'</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">VITE_GIT_COMMIT_HASH</span> <span class="o">=</span> <span class="nx">git</span><span class="p">.</span><span class="nx">short</span><span class="p">()</span></code></pre></div>
<p>Then add this to your Vue component to use it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="ts"><span class="kd">const</span> <span class="nx">gitHash</span> <span class="o">=</span> <span class="k">import</span><span class="p">.</span><span class="nx">meta</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">VITE_GIT_COMMIT_HASH</span></code></pre></div>
<p>then you can output it in templates etc as normal:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="handlebars"><span class="nt"><p></span><span class="k">{{</span> <span class="nv">gitHash</span> <span class="k">}}</span><span class="nt"></p></span></code></pre></div>Thanks, Andrew Gallant (burntsushi)!2023-05-22T13:29:10-07:002023-06-03T11:08:28-07:00Duncan Locktag:duncanlock.net,2023-05-22:/blog/2023/05/22/thanks-andrew-gallant-burntsushi/<p>Open source developers are often the unsung heroes of the technology world, creating <span class="amp">&</span> maintaining the software that powers our digital lives. Sadly, their contributions often go unnoticed, but without their dedication and expertise, much of the software <span class="amp">&</span> digital infrastructure what we take for granted today wouldn’t exist.</p>
<p>I want to recognize the awesome work of one developer and express my gratitude to them for making the world a better place through open source.</p>
<section class="doc-section level-1"><h2 id="_andrew_gallant_burntsushi">Andrew Gallant / burntsushi</h2><section class="doc-section level-2"><h3 id="_github_profile"><a href="https://github.com/burntsushi">GitHub Profile</a></h3><div class="quote-block"><blockquote><p>I love to code.</p></blockquote></div>
<figure class="image-block"><a class="image" href="https://github.com/burntsushi"><img src="https://avatars.githubusercontent.com/u/456674?v=4" alt="Avatar" width="100"></a>
<figcaption><a href="https://github.com/burntsushi">GitHub Profile</a></figcaption></figure>
<div class="ulist"><ul><li>Location: Marlborough, <span class="caps">MA</span></li><li>Followers: 7905</li><li><a href="https://github.com/burntsushi?tab=repositories&type=source">Public Repositories</a>: 156</li><li>@salesforce</li><li><a class="bare" href="https://burntsushi.net">https://burntsushi.net</a></li></ul></div>
<p class="lead clear">These are the projects of theirs that I love <span class="amp">&</span> use the most:</p></section>
<section class="doc-section level-2"><h3 id="_ripgrep"><a href="https://github.com/BurntSushi/ripgrep">ripgrep</a></h3><div class="quote-block"><blockquote><p>ripgrep recursively searches directories for a regex pattern while respecting your gitignore</p></blockquote></div>
<p>Ripgrep is a <em>superfast</em> replacement for grep, ag, ack etc…​ It’s not just <em>way</em> faster than all …</p></section></section><p>Open source developers are often the unsung heroes of the technology world, creating <span class="amp">&</span> maintaining the software that powers our digital lives. Sadly, their contributions often go unnoticed, but without their dedication and expertise, much of the software <span class="amp">&</span> digital infrastructure what we take for granted today wouldn’t exist.</p>
<p>I want to recognize the awesome work of one developer and express my gratitude to them for making the world a better place through open source.</p>
<section class="doc-section level-1"><h2 id="_andrew_gallant_burntsushi">Andrew Gallant / burntsushi</h2><section class="doc-section level-2"><h3 id="_github_profile"><a href="https://github.com/burntsushi">GitHub Profile</a></h3><div class="quote-block"><blockquote><p>I love to code.</p></blockquote></div>
<figure class="image-block"><a class="image" href="https://github.com/burntsushi"><img src="https://avatars.githubusercontent.com/u/456674?v=4" alt="Avatar" width="100"></a>
<figcaption><a href="https://github.com/burntsushi">GitHub Profile</a></figcaption></figure>
<div class="ulist"><ul><li>Location: Marlborough, <span class="caps">MA</span></li><li>Followers: 7905</li><li><a href="https://github.com/burntsushi?tab=repositories&type=source">Public Repositories</a>: 156</li><li>@salesforce</li><li><a class="bare" href="https://burntsushi.net">https://burntsushi.net</a></li></ul></div>
<p class="lead clear">These are the projects of theirs that I love <span class="amp">&</span> use the most:</p></section>
<section class="doc-section level-2"><h3 id="_ripgrep"><a href="https://github.com/BurntSushi/ripgrep">ripgrep</a></h3><div class="quote-block"><blockquote><p>ripgrep recursively searches directories for a regex pattern while respecting your gitignore</p></blockquote></div>
<p>Ripgrep is a <em>superfast</em> replacement for grep, ag, ack etc…​ It’s not just <em>way</em> faster than all of them, it also has full Unicode support and better defaults: recursive, and ignores files in any .gitignore files that it comes across. These are usually what I want, so no aliases, less typing and faster results. It’s written in Rust, so just download the single static binary and run it - and, if for some reason you’re forced to use it, Ripgrep also works on Windows.</p></section>
<section class="doc-section level-2"><h3 id="_xsv"><a href="https://github.com/BurntSushi/xsv">xsv</a></h3><div class="quote-block"><blockquote><p>A fast <span class="caps">CSV</span> command line toolkit written in Rust.</p></blockquote></div>
<p><span class="caps">XSV</span> is a <em>very fast</em> command line utility for indexing, slicing, analyzing, splitting and joining <span class="caps">CSV</span> files. If you’re going to be importing the <span class="caps">CSV</span> into a <span class="caps">DB</span>, it’s very useful for figuring out the shape of the data contained in the file, as well as any problems. If you’re not, you can use it to do all that - as well a querying the data on the command line with simple <span class="caps">SQL</span> and <span class="caps">JOIN</span> support.</p>
<p><span class="caps">XSV</span>’s written in Rust, so just download the single static binary and run it - and, works on Linux, MacOS <span class="amp">&</span> Windows.</p>
<p>The <a href="https://github.com/BurntSushi/xsv#readme"><span class="caps">README</span></a> is great and full of examples, but here’s a taster, showing the super useful <code>headers</code>, <code>count</code>, <code>stats</code>, <code>select</code>, <code>sample</code>, <code>join</code> <span class="amp">&</span> <code>table</code> commands:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">#</span><span class="w"> </span>Show the headers of a file:
<span class="gp">$</span><span class="w"> </span>xsv headers file.csv
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>Count the number of entries:
<span class="gp">$</span><span class="w"> </span>xsv count file.csv
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>Get an overview of the shape of entries:
<span class="gp">$</span><span class="w"> </span>xsv stats file.csv | xsv table
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>Select a few columns:
<span class="gp">$</span><span class="w"> </span>xsv <span class="k">select </span>column_a,column_b file.csv
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>Show 10 random entries
<span class="gp">$</span><span class="w"> </span>xsv sample 10 file.csv
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>Join a column from one file to another:
<span class="gp">$</span><span class="w"> </span>xsv <span class="nb">join</span> <span class="nt">--no-case</span> column_a file_a.csv column_b file_b.csv | xsv table</code></pre></div></section></section>Fixing apt: Key is stored in legacy trusted.gpg keyring Warnings2022-12-03T07:56:18-08:002022-12-03T07:56:18-08:00Duncan Locktag:duncanlock.net,2022-12-03:/blog/2022/12/03/fixing-apt-key-is-stored-in-legacy-trustedgpg-keyring-warnings/<p>If you see messages like this when running <span class="caps">APT</span> updated on Debian/Ubuntu systems:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="plain">W: https://download.virtualbox.org/virtualbox/debian/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.</code></pre></div>
<p>This is because using apt-key - particularly the mechanism where it dumps all its keys into one large file (/etc/apt/trusted.gpg) - is a bad idea and deprecated. Apt-Key trusts all those keys, for anything that apt is doing - not just for the particular repository that they key belongs to - anything!</p>
<p>Afaik, the only way to fix this is currently by hand.</p>
<p>The correct way to do that is as follows:</p>
<section class="doc-section level-1"><h2 id="_find_the_particular_key_that_you_want_to_fix">Find the particular key that you want to fix</h2><p>You probably have more than one, so dump the list:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-key list <span class="o">></span> ~/apt-key-list</code></pre></div>
<p>Then …</p></section><p>If you see messages like this when running <span class="caps">APT</span> updated on Debian/Ubuntu systems:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="plain">W: https://download.virtualbox.org/virtualbox/debian/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.</code></pre></div>
<p>This is because using apt-key - particularly the mechanism where it dumps all its keys into one large file (/etc/apt/trusted.gpg) - is a bad idea and deprecated. Apt-Key trusts all those keys, for anything that apt is doing - not just for the particular repository that they key belongs to - anything!</p>
<p>Afaik, the only way to fix this is currently by hand.</p>
<p>The correct way to do that is as follows:</p>
<section class="doc-section level-1"><h2 id="_find_the_particular_key_that_you_want_to_fix">Find the particular key that you want to fix</h2><p>You probably have more than one, so dump the list:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-key list <span class="o">></span> ~/apt-key-list</code></pre></div>
<p>Then grep the list:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">grep</span> <span class="nt">--context</span><span class="o">=</span>5 virtualbox ~/apt-key-list
<span class="go">
/etc/apt/trusted.gpg
--------------------
pub dsa1024 2010-05-18 [SC]
7B0F AB3A 13B9 0743 5925 D9C9 5442 2A4B 98AB 5139
</span><span class="gp">uid [ unknown] Oracle Corporation (VirtualBox archive signing key) <info@virtualbox.org></span><span class="w">
</span><span class="go">sub elg2048 2010-05-18 [E]
</span><span class="c">...
</span><span class="go">
pub rsa4096 2016-04-22 [SC]
B9F8 D658 297A F3EF C18D 5CDF A2F6 83C5 2980 AECF
</span><span class="gp">uid [ unknown] Oracle Corporation (VirtualBox archive signing key) <info@virtualbox.org></span><span class="w">
</span><span class="go">sub rsa4096 2016-04-22 [E]
</span></code></pre></div>
<p>or you can run apt-key list each time:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-key list | <span class="nb">grep </span>virtualbox</code></pre></div></section>
<section class="doc-section level-1"><h2 id="_export_the_key_to_a_new_file">Export the key to a new file</h2><p>The key id is the last 9 chars of the fingerprint, with the space removed, so: <code>98AB 5139</code> → <code>98AB5139</code>.
We can then use that to export the key to a new file <span class="amp">&</span> remove the old one from apt-key:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-key <span class="nb">export </span>98AB5139 | <span class="nb">sudo </span>gpg <span class="nt">--dearmour</span> <span class="nt">-o</span> /usr/share/keyrings/virtualbox.gpg
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-key del 98AB5139</code></pre></div>
<p>I chose to put these into <code>/usr/share/keyrings/</code>, which seems to be the “default” location.</p></section>
<section class="doc-section level-1"><h2 id="_update_the_apt_sourcelist_file_to_point_to_the_new_key">Update the apt source/list file to point to the new key</h2><p>All my existing apt repos <span class="amp">&</span> PPAs were configured using .list files in <code>/etc/apt/sources.list.d/</code>. These are tiny text files with one line per repo and look like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="deb">deb [arch=amd64] https://download.virtualbox.org/virtualbox/debian jammy contrib</code></pre></div>
<p>All options, like arch and key, are comma separated key/value pairs inside the <code>[]</code> after the initial <code>deb</code>.</p>
<p>Turns out there’s a “new” format (supported since apt 1.1, in 2015) for these, <a href="https://manpages.debian.org/stretch/apt/sources.list.5.en.html#DEB822-STYLE_FORMAT">deb-822 style</a> <code>.sources</code> files, which are multi-line and look like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="deb822"># VirtualBox Official Repo
Types: deb
Architectures: amd64
URIs: https://download.virtualbox.org/virtualbox/debian
Suites: jammy
Components: contrib
Signed-By: /usr/share/keyrings/virtualbox.org.gpg</code></pre></div>
<p>These obviously should have just been <code>.toml</code> files, but what can you do. They’re easier to read and write than the previous single line format.</p>
<p>You use the <code>Signed-By: /usr/share/keyrings/virtualbox.org.gpg</code> part to point to a key for that repo, which will be used by apt just for this purpose.</p></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><p>Pieced together from the following sources:</p>
<div class="ulist"><ul><li><a class="bare" href="https://repolib.readthedocs.io/en/latest/deb822-format.html">https://repolib.readthedocs.io/en/latest/deb822-format.html</a></li><li><a class="bare" href="https://unix.stackexchange.com/a/582853/41614">https://unix.stackexchange.com/a/582853/41614</a></li><li><a class="bare" href="https://askubuntu.com/a/605017/10015">https://askubuntu.com/a/605017/10015</a></li><li><a class="bare" href="https://askubuntu.com/a/1307181/10015">https://askubuntu.com/a/1307181/10015</a></li><li><a class="bare" href="https://askubuntu.com/a/1409985/10015">https://askubuntu.com/a/1409985/10015</a></li></ul></div></section>Automatically Publishing a Blogroll from an OPML File2022-11-22T20:50:49-08:002022-11-22T20:50:49-08:00Duncan Locktag:duncanlock.net,2022-11-22:/blog/2022/11/22/automatically-publishing-a-blogroll-from-an-opml-file/<p>I run a <a href="https://miniflux.app/">Miniflux</a> instance on my desktop computer, which fetches all my feeds and makes the content available locally.</p>
<p>Inspired by a <a href="https://news.ycombinator.com/item?id=33585201">discussion the other day on HackerNews</a>, I wrote a little script that asks Miniflux for a list of my feeds in <a href="https://en.wikipedia.org/wiki/OPML"><span class="caps">OPML</span></a> format and turns it into an AsciiDoc page, which I publish on here, as <a href="https://duncanlock.net/pages/blogroll-links.html">my BlogRoll <span class="amp">&</span> Links page</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="kn">import</span> <span class="nn">xml.etree.ElementTree</span> <span class="k">as</span> <span class="n">ET</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">[</span><span class="s">"MINIFLUX_API_KEY"</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">key</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"$MINIFLUX_API_KEY not set."</span><span class="p">)</span>
<span class="n">sys</span><span class="p">.</span><span class="nb">exit</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">url</span> <span class="o">=</span> <span class="s">"http://miniflux.home/v1/export"</span>
<span class="n">hdr</span> <span class="o">=</span> <span class="p">{</span><span class="s">"X-Auth-Token"</span><span class="p">:</span> <span class="n">key</span><span class="p">}</span>
<span class="n">request</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">hdr</span><span class="p">)</span>
<span class="n">opml</span> <span class="o">=</span> <span class="n">ET</span><span class="p">.</span><span class="n">fromstring</span><span class="p">(</span><span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">).</span><span class="n">read</span><span class="p">())</span>
<span class="n">updated</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">now</span><span class="p">().</span><span class="n">astimezone</span><span class="p">().</span><span class="n">replace</span><span class="p">(</span><span class="n">microsecond</span><span class="o">=</span><span class="mi">0</span><span class="p">).</span><span class="n">isoformat</span><span class="p">(</span><span class="s">" "</span><span class="p">)</span>
<span class="n">header</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"""
:title: Blogroll & Links
:slug: blogroll-links
:created: 2022-11-15 13 …</span></code></pre></div><p>I run a <a href="https://miniflux.app/">Miniflux</a> instance on my desktop computer, which fetches all my feeds and makes the content available locally.</p>
<p>Inspired by a <a href="https://news.ycombinator.com/item?id=33585201">discussion the other day on HackerNews</a>, I wrote a little script that asks Miniflux for a list of my feeds in <a href="https://en.wikipedia.org/wiki/OPML"><span class="caps">OPML</span></a> format and turns it into an AsciiDoc page, which I publish on here, as <a href="https://duncanlock.net/pages/blogroll-links.html">my BlogRoll <span class="amp">&</span> Links page</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">urllib.request</span>
<span class="kn">import</span> <span class="nn">xml.etree.ElementTree</span> <span class="k">as</span> <span class="n">ET</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">[</span><span class="s">"MINIFLUX_API_KEY"</span><span class="p">]</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">key</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">"$MINIFLUX_API_KEY not set."</span><span class="p">)</span>
<span class="n">sys</span><span class="p">.</span><span class="nb">exit</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">url</span> <span class="o">=</span> <span class="s">"http://miniflux.home/v1/export"</span>
<span class="n">hdr</span> <span class="o">=</span> <span class="p">{</span><span class="s">"X-Auth-Token"</span><span class="p">:</span> <span class="n">key</span><span class="p">}</span>
<span class="n">request</span> <span class="o">=</span> <span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">Request</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">hdr</span><span class="p">)</span>
<span class="n">opml</span> <span class="o">=</span> <span class="n">ET</span><span class="p">.</span><span class="n">fromstring</span><span class="p">(</span><span class="n">urllib</span><span class="p">.</span><span class="n">request</span><span class="p">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">request</span><span class="p">).</span><span class="n">read</span><span class="p">())</span>
<span class="n">updated</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">now</span><span class="p">().</span><span class="n">astimezone</span><span class="p">().</span><span class="n">replace</span><span class="p">(</span><span class="n">microsecond</span><span class="o">=</span><span class="mi">0</span><span class="p">).</span><span class="n">isoformat</span><span class="p">(</span><span class="s">" "</span><span class="p">)</span>
<span class="n">header</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"""
:title: Blogroll & Links
:slug: blogroll-links
:created: 2022-11-15 13:23:54-08:00
:date: </span><span class="si">{</span><span class="n">updated</span><span class="si">}</span><span class="s">
:meta_description: These are the feeds that I read, exported from my home Miniflux instance as an OPML file & converted to AsciiDoc.
:toc:
:toc-class: page-toc
[.lead]
These are the feeds that I read, exported from my home https://miniflux.app/[Miniflux] instance as an https://en.wikipedia.org/wiki/OPML[OPML] file & converted to AsciiDoc.
See how this page is made: link:++{{filename}}/posts/tech/automatically-publishing-a-blogroll-from-an-opml-file.adoc++[Automatically Publishing a Blogroll from an OPML File]
"""</span>
<span class="k">print</span><span class="p">(</span><span class="n">header</span><span class="p">)</span>
<span class="n">feed_icon</span> <span class="o">=</span> <span class="s">'<i class="icon"><svg><use href="#feed"></use></svg></i>'</span>
<span class="k">for</span> <span class="n">section</span> <span class="ow">in</span> <span class="n">opml</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span>
<span class="k">if</span> <span class="n">section</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"text"</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">[</span><span class="s">"personal"</span><span class="p">]:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">'</span><span class="se">\n</span><span class="s">## </span><span class="si">{</span><span class="n">section</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"text"</span><span class="p">].</span><span class="n">replace</span><span class="p">(</span><span class="s">"-"</span><span class="p">,</span> <span class="s">" "</span><span class="p">).</span><span class="n">title</span><span class="p">()</span><span class="si">}</span><span class="se">\n</span><span class="s">'</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"[.three-columns]"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">link</span> <span class="ow">in</span> <span class="n">section</span><span class="p">:</span>
<span class="n">feed_link</span> <span class="o">=</span> <span class="sa">f</span><span class="s">'+++<a href="</span><span class="si">{</span><span class="n">link</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"xmlUrl"</span><span class="p">]</span><span class="si">}</span><span class="s">" title="Link to Feed"></span><span class="si">{</span><span class="n">feed_icon</span><span class="si">}</span><span class="s"></a>+++'</span>
<span class="k">print</span><span class="p">(</span>
<span class="sa">f</span><span class="s">'* </span><span class="si">{</span><span class="n">link</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"htmlUrl"</span><span class="p">]</span><span class="si">}</span><span class="s">[</span><span class="si">{</span><span class="n">link</span><span class="p">.</span><span class="n">attrib</span><span class="p">[</span><span class="s">"title"</span><span class="p">]</span><span class="si">}</span><span class="s">,title="Link to Website"] </span><span class="si">{</span><span class="n">feed_link</span><span class="si">}</span><span class="s">'</span>
<span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">'''</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Last updated: </span><span class="si">{</span><span class="n">updated</span><span class="si">}</span><span class="s">.</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span></code></pre></div>
<p>This outputs to stdout and expects to be run like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>python ./opml2adoc.py <span class="o">></span> content/pages/blogroll-links.adoc</code></pre></div>Review: Morcheeba Blackest Blue Tour, Vancouver 20222022-10-18T20:00:00-08:002022-10-18T20:00:00-08:00Duncan Locktag:duncanlock.net,2022-10-18:/blog/2022/10/18/review-morcheeba-blackest-blue-tour-vancouver-2022/<p class="lead">Music great, lighting great, encore fantastic. Loads of audience participation, singing and dancing throughout. Fun!</p>
<p>I don’t particularly like the Commodore Ballroom, as a venue. It’s <span class="caps">OK</span>, but it’s a bit of an echoey box. The sound for Loviet, the warm-up band, was a bit echoey and too loud, lyrics indistinct.</p>
<p>They pulled it together though and the sound for Morcheeba was <em>great</em>. I was about four people from the front, great view <span class="amp">&</span> atmosphere - although I seemed to be in the unofficial tall guys section for some of the gig.</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/Fr8PwCx-9R4?rel=0" frameborder="0" allowfullscreen></iframe></div>
<p>The band seemed to be having a good time - just like the audience. They were all excellent, from Skye Edwards huge stage presence and amazing voice, to Ross Godfrey’s great guitar work. It was lots of fun, music was great, lighting was great, encore was fantastic …</p><p class="lead">Music great, lighting great, encore fantastic. Loads of audience participation, singing and dancing throughout. Fun!</p>
<p>I don’t particularly like the Commodore Ballroom, as a venue. It’s <span class="caps">OK</span>, but it’s a bit of an echoey box. The sound for Loviet, the warm-up band, was a bit echoey and too loud, lyrics indistinct.</p>
<p>They pulled it together though and the sound for Morcheeba was <em>great</em>. I was about four people from the front, great view <span class="amp">&</span> atmosphere - although I seemed to be in the unofficial tall guys section for some of the gig.</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/Fr8PwCx-9R4?rel=0" frameborder="0" allowfullscreen></iframe></div>
<p>The band seemed to be having a good time - just like the audience. They were all excellent, from Skye Edwards huge stage presence and amazing voice, to Ross Godfrey’s great guitar work. It was lots of fun, music was great, lighting was great, encore was fantastic. Loads of audience participation, singing and dancing throughout.</p>
<section class="doc-section level-1"><h2 id="_morcheeba_tour_setlist_vancouver_2022"><a href="https://music.youtube.com/playlist?list=PLxMjPyxIGFZTW9K-9_jmtdX8kv8pPQTwz">Morcheeba Tour Setlist, Vancouver 2022</a></h2><p>October 18th, at The Commodore Ballroom, Vancouver, <span class="caps">BC</span>, Canada</p>
<div class="table-block scrollable"><table class="frame-all grid-all stretch"><colgroup><col style="width: 37.5%;"><col style="width: 25%;"><col style="width: 25%;"><col style="width: 12.5%;"></colgroup><thead><tr><th class="halign-left valign-top">Title</th><th class="halign-left valign-top">Artist</th><th class="halign-left valign-top">Album</th><th class="halign-left valign-top">Duration</th></tr></thead><tbody><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/jBB1DppZrbazEN8WkCBzN-m7JKiUnfS1hZZNv6XAkqqPkYviyfyVOcfL0ayTNlAQc4Z2MabmkMHHS5NsaA=w60-h60-s-l90-rj" alt="thumbnail" width="40" height="40" class="bare">The Sea</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Big Calm</td><td class="halign-left valign-top">5:48</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/jBB1DppZrbazEN8WkCBzN-m7JKiUnfS1hZZNv6XAkqqPkYviyfyVOcfL0ayTNlAQc4Z2MabmkMHHS5NsaA=w60-h60-s-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Friction</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Big Calm</td><td class="halign-left valign-top">4:15</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/GopTaNjmiO0ACJYVrK9X1pz5GINLMtiioWTqktW3I7YuvFTehvI713wCiJ6-5D7aC_8pd_j5ZlAjCZQ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Otherwise</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Charango</td><td class="halign-left valign-top">3:43</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kXisPfZHqSILydvv_aDP_BhROCHNWfuYGxRyIBj7mPRjKFO8h9xQ5yrBnjAItVt-hqu8KxYoaOQitQ7YJQ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Never an Easy Way</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Who Can You Trust?</td><td class="halign-left valign-top">6:43</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/JoHyOvRSKAeZH3cuo7FwS-y5UPLUSFvMlYf7kEf1k2Yds1GQSaBhzx3iRreUuzAZsTqz3DU9V8onqvS3wQ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Sounds Of Blue</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Blackest Blue</td><td class="halign-left valign-top">3:35</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/jBB1DppZrbazEN8WkCBzN-m7JKiUnfS1hZZNv6XAkqqPkYviyfyVOcfL0ayTNlAQc4Z2MabmkMHHS5NsaA=w60-h60-s-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Part of the Process</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Big Calm</td><td class="halign-left valign-top">4:26</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kXisPfZHqSILydvv_aDP_BhROCHNWfuYGxRyIBj7mPRjKFO8h9xQ5yrBnjAItVt-hqu8KxYoaOQitQ7YJQ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Trigger Hippie</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Who Can You Trust?</td><td class="halign-left valign-top">5:32</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/JoHyOvRSKAeZH3cuo7FwS-y5UPLUSFvMlYf7kEf1k2Yds1GQSaBhzx3iRreUuzAZsTqz3DU9V8onqvS3wQ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Oh Oh Yeah</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Blackest Blue</td><td class="halign-left valign-top">6:54</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ATXMPmvPaxp_YGNc6u9nHR68mBQmgaqyD2pxdFX5UXgSdqLnu97zG-O8wOK-U_d83OQBDO3qw0xY7qg=w60-h60-s-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Blood Like Lemonade</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Blood Like Lemonade</td><td class="halign-left valign-top">4:54</td></tr><tr><td class="halign-left valign-top"><img src="https://i.ytimg.com/vi/jFqpk3Miask/sddefault.jpg?sqp=-oaymwEWCJADEOEBIAQqCghqEJQEGHgg6AJIWg&rs=AMzJL3kmYdq0Bq51oG2LvJrxlt4cUqf2Fg" alt="thumbnail" width="40" height="40" class="bare">Morcheeba - Get along @ Montreux Jazz Festival 2003</td><td class="halign-left valign-top">bergkoen</td><td class="halign-left valign-top"></td><td class="halign-left valign-top">3:39</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/jBB1DppZrbazEN8WkCBzN-m7JKiUnfS1hZZNv6XAkqqPkYviyfyVOcfL0ayTNlAQc4Z2MabmkMHHS5NsaA=w60-h60-s-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Let Me See</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Big Calm</td><td class="halign-left valign-top">4:21</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/sAVfyzrUyQAp0GrA38HAqC50cbREm-I5C-jSYbR5CcFZzXNPZJryCBLx79pBcA97oriPB96GCLIl3aHI=w60-h60-s-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Rome Wasn’t Built in a Day</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Rome Wasn’t Built In A Day</td><td class="halign-left valign-top">3:35</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/N-z4qv82-tTH07zw3nJbtU4To-3andLthasZdj0P9ujW3Tif7xAVRwr4k2x4CsH37piCCs3-jLM8V5QQ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Sweet <span class="caps">L.A.</span></td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Blaze Away</td><td class="halign-left valign-top">2:41</td></tr><tr><td class="halign-left valign-top"><img src="https://i.ytimg.com/vi/RtS1fP6lsmo/sddefault.jpg?sqp=-oaymwEWCJADEOEBIAQqCghqEJQEGHgg6AJIWg&rs=AMzJL3lUPd1Kl-06yrSV8T54sxAk7Pvb4A" alt="thumbnail" width="40" height="40" class="bare">morcheeba don’t let it bring you down live @ fnac</td><td class="halign-left valign-top">eris x nyx</td><td class="halign-left valign-top"></td><td class="halign-left valign-top">3:49</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/N4dEx8GybWevR_Ir3sdBm6HHhvBqKKBJv_MwhKv3gB9326wJcElBIlMSul7mKuZBJHg3ghA94KtCN1Qi=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Enjoy The Ride (feat. Judie Tzuke)</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Dive Deep</td><td class="halign-left valign-top">4:01</td></tr><tr><td class="halign-left valign-top"><img src="https://i.ytimg.com/vi/fIMXXUcCRaQ/sddefault.jpg?sqp=-oaymwEWCJADEOEBIAQqCghqEJQEGHgg6AJIWg&rs=AMzJL3npTJFDly0mZPfhjddiruscst-9jg" alt="thumbnail" width="40" height="40" class="bare">Morcheeba live in Munich ‘22 - Gimmie Shelter</td><td class="halign-left valign-top">ilrub64</td><td class="halign-left valign-top"></td><td class="halign-left valign-top">4:00</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/jBB1DppZrbazEN8WkCBzN-m7JKiUnfS1hZZNv6XAkqqPkYviyfyVOcfL0ayTNlAQc4Z2MabmkMHHS5NsaA=w60-h60-s-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Blindfold</td><td class="halign-left valign-top">Morcheeba</td><td class="halign-left valign-top">Big Calm</td><td class="halign-left valign-top">4:39</td></tr></tbody></table></div>
<hr>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/review-morcheeba-blackest-blue-tour-vancouver-2022/PXL_20221019_052330003.MP.avif" alt="Photo from where I was standing, showing Skye Edwards centre stage, singing, wearing a billowy dress and tall hat. Behing her you can see the drummer and keyboardist.">
<figcaption>Figure 1. My view of the stage - not quite at the front, but only a few people back. This round thing at the bottom is a bladelss fan, used for extra billowyness.</figcaption></figure></section>Review: Florence + the Machine Gig, Vancouver 20222022-10-04T20:00:00-08:002023-05-24T09:42:14-07:00Duncan Locktag:duncanlock.net,2022-10-04:/blog/2022/10/04/review-florence-the-machine-gig-vancouver-2022/<p class="lead">Just experienced the incredible, joyous, wonderful Florence + the Machine in Vancouver. Amazing. I’m officially part of Miss Haversham’s cult now!</p>
<p>I’ve been a fan for ages; their music just really lifts me up. This is the first time I’ve seen them live, and they really blew me away - what a great show!</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/k_reJGGgYCg?rel=0" frameborder="0" allowfullscreen></iframe></div>
<p>This was their first tour post-covid, so included some emotional tales about how the pandemic affected people who’s vocation - and living - comes from performing live music.</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/OEvq8qc15p8?rel=0" frameborder="0" allowfullscreen></iframe></div>
<section class="doc-section level-1"><h2 id="_learnings">Learnings</h2><p>Things I learned from this gig:</p>
<div class="ulist"><ul><li>Rogers Arena is a pretty good venue, especially if you’re on the floor, near the front.</li><li>Make a playlist after each gig with the setlist, so that you can relive the gig. I’ve been making these on YouTube Music, usually with the setlist from <a class="bare" href="https://www.setlist.fm/">https://www.setlist.fm …</a></li></ul></div></section><p class="lead">Just experienced the incredible, joyous, wonderful Florence + the Machine in Vancouver. Amazing. I’m officially part of Miss Haversham’s cult now!</p>
<p>I’ve been a fan for ages; their music just really lifts me up. This is the first time I’ve seen them live, and they really blew me away - what a great show!</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/k_reJGGgYCg?rel=0" frameborder="0" allowfullscreen></iframe></div>
<p>This was their first tour post-covid, so included some emotional tales about how the pandemic affected people who’s vocation - and living - comes from performing live music.</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/OEvq8qc15p8?rel=0" frameborder="0" allowfullscreen></iframe></div>
<section class="doc-section level-1"><h2 id="_learnings">Learnings</h2><p>Things I learned from this gig:</p>
<div class="ulist"><ul><li>Rogers Arena is a pretty good venue, especially if you’re on the floor, near the front.</li><li>Make a playlist after each gig with the setlist, so that you can relive the gig. I’ve been making these on YouTube Music, usually with the setlist from <a class="bare" href="https://www.setlist.fm/">https://www.setlist.fm/</a> and my memory. I really <em>love</em> these playlists - it’s such a simple thing to do, but I’ve found that it really keeps the hugely uplifting positive feelings I get from live music going all week!</li></ul></div></section>
<section class="doc-section level-1"><h2 id="_florence_the_machine_dance_fever_tour_setlist_vancouver_2022"><a href="https://music.youtube.com/playlist?list=PLxMjPyxIGFZQB1GY-dG424ylRh6_XZaBm">Florence + the Machine Dance Fever Tour Setlist, Vancouver 2022</a></h2><p>October 4th, at Rogers Arena, Vancouver, <span class="caps">BC</span>, Canada</p>
<div class="table-block scrollable"><table class="frame-all grid-all stretch"><colgroup><col style="width: 37.5%;"><col style="width: 25%;"><col style="width: 25%;"><col style="width: 12.5%;"></colgroup><thead><tr><th class="halign-left valign-top">Title</th><th class="halign-left valign-top">Artist</th><th class="halign-left valign-top">Album</th><th class="halign-left valign-top">Duration</th></tr></thead><tbody><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Heaven Is Here</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">1:52</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">King</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">4:41</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/JoknLykZgo9MPvGARbYOoZwBFsUVeY_ayELBIe6a53y1u7YsWptSypiuSYTrIDzAtfhwNCDMwzMlxx-K=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Ship To Wreck</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">How Big, How Blue, How Beautiful</td><td class="halign-left valign-top">3:54</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Free</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">3:55</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Daffodil</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">3:35</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/af5hg1bnRhz5rISIoIxxskutD78hUTDQfo_5q1-5qbz4FQrcyJ7OaP-_If2GuI9Li5AZITFlVgyO0aDJ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Dog Days Are Over</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Lungs (Digital Deluxe Version)</td><td class="halign-left valign-top">4:13</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Girls Against God</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">4:41</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Dream Girl Evil</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">3:48</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Prayer Factory</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">1:14</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/3U-Wh9ghTJgqAhrq6peZ0dr26QF0e49JxEGnJw7SIg3fKpR-TuydwBJmhu7BZbeKDLt4ZaZyDVchs7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Big God</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">High As Hope</td><td class="halign-left valign-top">4:02</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Cassandra</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">4:18</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/JoknLykZgo9MPvGARbYOoZwBFsUVeY_ayELBIe6a53y1u7YsWptSypiuSYTrIDzAtfhwNCDMwzMlxx-K=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">What Kind Of Man</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">How Big, How Blue, How Beautiful</td><td class="halign-left valign-top">3:36</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Morning Elvis</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">4:23</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/3U-Wh9ghTJgqAhrq6peZ0dr26QF0e49JxEGnJw7SIg3fKpR-TuydwBJmhu7BZbeKDLt4ZaZyDVchs7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">June</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">High As Hope</td><td class="halign-left valign-top">3:42</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/3U-Wh9ghTJgqAhrq6peZ0dr26QF0e49JxEGnJw7SIg3fKpR-TuydwBJmhu7BZbeKDLt4ZaZyDVchs7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Hunger</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">High As Hope</td><td class="halign-left valign-top">3:35</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Choreomania</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">3:34</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/af5hg1bnRhz5rISIoIxxskutD78hUTDQfo_5q1-5qbz4FQrcyJ7OaP-_If2GuI9Li5AZITFlVgyO0aDJ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Kiss With A Fist</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Lungs (International Version)</td><td class="halign-left valign-top">2:04</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/af5hg1bnRhz5rISIoIxxskutD78hUTDQfo_5q1-5qbz4FQrcyJ7OaP-_If2GuI9Li5AZITFlVgyO0aDJ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Cosmic Love</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Lungs (Digital Deluxe Version)</td><td class="halign-left valign-top">4:16</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">My Love</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">3:52</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/ZXUTwA-JkSXUGrLZ2vlbLXc5KKbOcQlhF8mhmommDegKGLkPEZDNPlzjLZ2hhodrbr9PWOlqtIY1B7g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Restraint</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Dance Fever</td><td class="halign-left valign-top">0:48</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/KYNuXSbaHS9dt2uwyzErjNiQ69_7EeFQk8ITYjGssGfk0z-6BJEwc2J6gN5Z2tCH533nKd45aMmHEVS1=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Never Let Me Go</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Ceremonials (Deluxe Edition)</td><td class="halign-left valign-top">4:32</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/KYNuXSbaHS9dt2uwyzErjNiQ69_7EeFQk8ITYjGssGfk0z-6BJEwc2J6gN5Z2tCH533nKd45aMmHEVS1=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Shake It Out</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Ceremonials (Deluxe Edition)</td><td class="halign-left valign-top">4:38</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/af5hg1bnRhz5rISIoIxxskutD78hUTDQfo_5q1-5qbz4FQrcyJ7OaP-_If2GuI9Li5AZITFlVgyO0aDJ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Rabbit Heart (Raise It Up)</td><td class="halign-left valign-top">Florence + The Machine</td><td class="halign-left valign-top">Lungs (Digital Deluxe Version)</td><td class="halign-left valign-top">3:53</td></tr></tbody></table></div></section>Supply Chain Attacks & Package Managers - a Solution?2022-05-29T20:35:02-07:002022-06-01T09:52:54-07:00Duncan Locktag:duncanlock.net,2022-05-29:/blog/2022/05/29/supply-chain-attacks-package-managers-a-solution/<p>In this <a href="https://drewdevault.com/2022/05/12/Supply-chain-when-will-we-learn.html">When Will We Learn</a> post, Drew DeVault talks about supply chain attacks against language package managers (npm, PyPI, cargo, etc…​) - and compares them to official Linux distribution repositories (deb, rpm, etc…​).</p>
<p>The conclusion drawn was:</p>
<div class="quote-block"><blockquote><p>The correct way to ship packages is with your distribution’s package manager. These have a separate review step, completely side-stepping typo-squatting, establishing a long-term relationship of trust between the vendor and the distribution packagers, and providing a dispassionate third-party to act as an intermediary between users and vendors. Furthermore, they offer stable distributions which can be relied upon for an extended period of time, provide cohesive whole-system integration testing, and unified patch distribution and <span class="caps">CVE</span> notifications for your entire system.</p><footer>— <cite>Drew DeVault, <a href="https://drewdevault.com/2022/05/12/Supply-chain-when-will-we-learn.html#why-is-this-happening">When Will We Learn</a></cite></footer></blockquote></div>
<p>I think I agree with this, essentially. We <em>do</em> need to change the way we do …</p><p>In this <a href="https://drewdevault.com/2022/05/12/Supply-chain-when-will-we-learn.html">When Will We Learn</a> post, Drew DeVault talks about supply chain attacks against language package managers (npm, PyPI, cargo, etc…​) - and compares them to official Linux distribution repositories (deb, rpm, etc…​).</p>
<p>The conclusion drawn was:</p>
<div class="quote-block"><blockquote><p>The correct way to ship packages is with your distribution’s package manager. These have a separate review step, completely side-stepping typo-squatting, establishing a long-term relationship of trust between the vendor and the distribution packagers, and providing a dispassionate third-party to act as an intermediary between users and vendors. Furthermore, they offer stable distributions which can be relied upon for an extended period of time, provide cohesive whole-system integration testing, and unified patch distribution and <span class="caps">CVE</span> notifications for your entire system.</p><footer>— <cite>Drew DeVault, <a href="https://drewdevault.com/2022/05/12/Supply-chain-when-will-we-learn.html#why-is-this-happening">When Will We Learn</a></cite></footer></blockquote></div>
<p>I think I agree with this, essentially. We <em>do</em> need to change the way we do dependencies when developing - and having someone else review packages would help reduce supply chain attacks.</p>
<p>I wanted to try and figure out if this solution - use official Linux distribution packages instead of language ones - would work in practice, what that might look like, and how that might scale.</p>
<aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>All the numbers used here have lots of obvious caveats (how active are all these volunteers?, how up to date, accurate?, etc…​) - but, these are the best numbers I could find in the time available, so let’s just assume they’re good enough for some order-of-magnitude/ballpark estimates.</p></aside>
<section class="doc-section level-1"><h2 id="_debian_vs_pypi">Debian vs PyPI</h2><p>There are lots of Linux distributions and lots of language package managers <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>:</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 20%;"><col style="width: 20%;"><col style="width: 20%;"><col style="width: 20%;"><col style="width: 20%;"></colgroup><thead><tr><th class="halign-left valign-top">Repository</th><th class="halign-left valign-top">Language</th><th class="halign-right valign-top">Package Count</th><th class="halign-right valign-top">Avg. Growth/day</th><th class="halign-center valign-top">3rd Party Vetting?</th></tr></thead><tbody><tr><td class="halign-left valign-top"><a href="https://npmjs.org/">npm</a></td><td class="halign-left valign-top">JavaScript</td><td class="halign-right valign-top">1,965,443</td><td class="halign-right valign-top">962</td><td class="halign-center valign-top">❌</td></tr><tr><td class="halign-left valign-top"><a href="https://www.maven.org/">Maven</a></td><td class="halign-left valign-top">Java</td><td class="halign-right valign-top">473,973</td><td class="halign-right valign-top">155</td><td class="halign-center valign-top">❌</td></tr><tr><td class="halign-left valign-top"><a href="https://pypi.org/">PyPI</a></td><td class="halign-left valign-top">Python</td><td class="halign-right valign-top">375,716</td><td class="halign-right valign-top">207</td><td class="halign-center valign-top">❌</td></tr><tr><td class="halign-left valign-top"><a href="https://rubygems.org/">RubyGems</a></td><td class="halign-left valign-top">Ruby</td><td class="halign-right valign-top">171,620</td><td class="halign-right valign-top">17</td><td class="halign-center valign-top">❌</td></tr><tr><td class="halign-left valign-top"><a href="https://crates.io/">Crates</a></td><td class="halign-left valign-top">Rust</td><td class="halign-right valign-top">83,189</td><td class="halign-right valign-top">75</td><td class="halign-center valign-top">❌</td></tr><tr><td class="halign-left valign-top"><a href="https://packages.debian.org/stable/">Debian</a></td><td class="halign-left valign-top">Debian Linux</td><td class="halign-right valign-top">96,728</td><td class="halign-right valign-top">?<a class="footnote-ref" id="_footnoteref_2" href="#_footnote_2" title="View footnote 2" role="doc-noteref">[2]</a></td><td class="halign-center valign-top">✅</td></tr><tr><td class="halign-left valign-top"><a href="https://aur.archlinux.org/packages">Arch <span class="caps">AUR</span></a></td><td class="halign-left valign-top">Arch Linux <span class="caps">AUR</span></td><td class="halign-right valign-top">74,694</td><td class="halign-right valign-top">26</td><td class="halign-center valign-top">❌</td></tr><tr><td class="halign-left valign-top"><a href="https://archlinux.org/packages">Arch</a></td><td class="halign-left valign-top">Arch Linux</td><td class="halign-right valign-top">13,006</td><td class="halign-right valign-top">?</td><td class="halign-center valign-top">✅</td></tr></tbody></table></div>
<p>I’m not going to try to compare them all - I’m going to pick two. I’m going start with <a href="https://www.debian.org/">Debian</a> - one of the largest open source collaborations in the world, and the Python package repo, <a href="https://pypi.org/">PyPI</a> - a medium-sized language package repo.</p>
<p>So, the Debian stable repo currently contains 96,728 packages - packages curated and maintained by dedicated 3rd party package maintainers. The PyPI repo contains 375,716 packages - uploaded by anyone, <em>usually</em> whoever wrote the Python module, but, really <em>anyone</em>. The python repo has roughly three times the number of packages as Debian.</p>
<aside class="sidebar"><p>If you want to look, you can download <a href="https://packages.debian.org/stable/allpackages?format=txt.gz">the Debian stable package list from here</a>.</p>
<p>There <em>are</em> actually some python packages in the Debian repo, that you can use apt to install, if you like. There are currently about 762 of them:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="shell">➜ <span class="nb">tail</span> <span class="nt">--lines</span> +7 allpackages.txt | <span class="nb">grep</span> ^python- | <span class="nb">wc</span> <span class="nt">-l</span>
762</code></pre></div>
<p>I think that these are <em>mostly</em> system utilities that are written in python <span class="amp">&</span> their dependencies, or just random python stuff that someone felt like getting into the Debian repo.</p></aside></section>
<section class="doc-section level-1"><h2 id="_what_would_pypi_debian_org_look_like">What would pypi.debian.org look like?</h2><p>From the point of view of a user, there are two major different between the Linux system package managers <span class="amp">&</span> the language ones:</p>
<section class="doc-section level-2"><h3 id="_system_wide">System Wide</h3><p>The first is that the system ones are (mostly) intended to install packages globally, at the system/<span class="caps">OS</span> level - and the language ones now mostly install into a folder/local virtual environment <a class="footnote-ref" id="_footnoteref_3" href="#_footnote_3" title="View footnote 3" role="doc-noteref">[3]</a>. This means that you can have an independent set of packages installed for each project that you’re working on. This avoids the version clashes/dll/dependency hell type stuff that happens if you have one global set of packages and is currently considered “best practice”, mostly.</p>
<p>So, a Debian python repo wouldn’t turn everything into .deb packages and use apt - not if you wanted anyone to use it. To get any traction with users, it would have to work the same way as upstream PyPI and work with the same tools <span class="amp">&</span> workflow - pip, poetry, etc…​ You’d just configure your tools to talk to pypi.debian.org, instead of pypi.org.</p>
<p>The <em>difference</em> would be that the packages are hosted by Debian and vetted/maintained by Debian package maintainers, like Debian stable deb packages are.</p></section>
<section class="doc-section level-2"><h3 id="_package_freshness">Package Freshness</h3><p>The second major difference is package freshness. Language package managers like PyPI have the very latest version of everything all the time. Developers publish new packages whenever they release new version of their packages, often completely automatically. The versions of packages in the Debian stable repo are fixed at release time, and only get urgent security fixes after that - hence the name “stable”. There’s also Debian Testing, which has more up-to-date packages, which will become the next stable repo when the next version of Debian is released. In general, language repo’s are always up-to-date and Debian repos are always behind.</p></section></section>
<section class="doc-section level-1"><h2 id="_how_many_maintainers_would_you_need">How many maintainers would you need?</h2><p>According to <a href="https://nm.debian.org/public/people/dm_all/">this list</a>, the Debian project currently has 240 maintainers. Given that Debian has 96,728 packages / 240 maintainers, that’s <strong>403 packages each</strong>.</p>
<aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>As a software developer, my initial reaction was there’s <em>no way</em> I’d be comfortable looking after 403 packages and vetting <span class="amp">&</span> code reviewing them - that…​ sounds like a lot! But is it really?</p>
<p>After the initial review, you’d only be reviewing incremental changes, so that would reduce the workload.</p>
<p>It seems likely that package updates follow a power law, so you’d have a few packages that update a lot, quickly falling to a long tail of packages that update almost never, so it might not be too bad after all?</p></aside>
<p>Anyway, if we just extrapolate those numbers to a Debian version of PyPI, that would mean that you’d need…​ 375,716 packages / 403 each = <strong>932 maintainers</strong> to run it. That’s quite a lot. The <em>entire Debian project</em> membership is currently <a href="https://nm.debian.org/members/">1022 people</a>.</p>
<p>So, if you wanted to maintain a Debian version of the python package repo, with roughly the same amount of package vetting as Debian stable, <strong><em>you’d need a volunteer effort about the same size as the whole Debian project, all over again</em></strong>.</p>
<p>You’d <em>also</em> need to add one new maintainer roughly every two days, to keep up with new package growth.</p>
<aside class="sidebar"><p>Just for fun, if you wanted to do npm, you’d need ~4877 maintainers for its 1,965,443 packages - and you’d need to <em>add 2 new maintainers every day</em>, just to keep up with growth.</p></aside>
<p>You can obviously argue these numbers, but whatever the Debian project is doing, they seem to have been doing it fairly successfully <a href="https://www.debian.org/doc/manuals/project-history/">since 1993</a>; whatever it is, it looks at least somewhat sustainable.</p>
<p>If you think about it, all these packages from all these different repositories are, roughly, the <em>output</em> of the open source ecosystem. If you want to get someone else, other than the developers, to review all this stuff, you are either going to need your existing volunteer developers to up their volunteer workload and review each other’s stuff - or you are going to need to get a load more volunteers from somewhere.</p>
<p>Anyway, that seems like a big ask - a lot of people. Maybe we could optimize this somehow - work smarter, not harder?</p></section>
<section class="doc-section level-1"><h2 id="_20_of_the_packages_80_of_the_value">20% of the packages, 80% of the value?</h2><p>Maybe we don’t need everything, just the popular stuff? How many packages account for 80% of downloads?</p>
<p>According to <a href="https://pypistats.org/packages/__all__">PyPI Stats</a>, PyPI had a total of 14,756,299,061 package downloads last month. <em>Fourteen billion package downloads per month</em> - that’s quite a lot! The most downloaded package was <a href="https://pypistats.org/packages/boto3">boto3</a>, with 325,102,697 downloads. So that package accounted for 2.2% of all downloads.</p>
<aside class="sidebar"><p>325 million is only 2%! Huh. Billions are really big!</p></aside>
<p>How about <a href="https://pypistats.org/top">the top 20 packages</a>?</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 33.3333%;"><col style="width: 33.3333%;"><col style="width: 33.3334%;"></colgroup><thead><tr><th class="halign-left valign-top">Package</th><th class="halign-right valign-top">Downloads</th><th class="halign-right valign-top">% of Total</th></tr></thead><tfoot><tr><td class="halign-left valign-top">Total</td><td class="halign-right valign-top">3,106,681,096</td><td class="halign-right valign-top">21.05%</td></tr></tfoot><tbody><tr><td class="halign-left valign-top">boto3</td><td class="halign-right valign-top">325,102,697</td><td class="halign-right valign-top">2.20%</td></tr><tr><td class="halign-left valign-top">urllib3</td><td class="halign-right valign-top">210,456,675</td><td class="halign-right valign-top">1.43%</td></tr><tr><td class="halign-left valign-top">botocore</td><td class="halign-right valign-top">207,095,211</td><td class="halign-right valign-top">1.40%</td></tr><tr><td class="halign-left valign-top">requests</td><td class="halign-right valign-top">200,489,161</td><td class="halign-right valign-top">1.36%</td></tr><tr><td class="halign-left valign-top">idna</td><td class="halign-right valign-top">172,283,921</td><td class="halign-right valign-top">1.17%</td></tr><tr><td class="halign-left valign-top">setuptools</td><td class="halign-right valign-top">168,960,136</td><td class="halign-right valign-top">1.15%</td></tr><tr><td class="halign-left valign-top">s3transfer</td><td class="halign-right valign-top">168,397,166</td><td class="halign-right valign-top">1.14%</td></tr><tr><td class="halign-left valign-top">typing-extensions</td><td class="halign-right valign-top">161,630,822</td><td class="halign-right valign-top">1.10%</td></tr><tr><td class="halign-left valign-top">six</td><td class="halign-right valign-top">152,703,179</td><td class="halign-right valign-top">1.03%</td></tr><tr><td class="halign-left valign-top">certifi</td><td class="halign-right valign-top">147,959,264</td><td class="halign-right valign-top">1.00%</td></tr><tr><td class="halign-left valign-top">python-dateutil</td><td class="halign-right valign-top">146,990,800</td><td class="halign-right valign-top">1.00%</td></tr><tr><td class="halign-left valign-top">pyyaml</td><td class="halign-right valign-top">138,941,619</td><td class="halign-right valign-top">0.94%</td></tr><tr><td class="halign-left valign-top">charset-normalizer</td><td class="halign-right valign-top">135,959,075</td><td class="halign-right valign-top">0.92%</td></tr><tr><td class="halign-left valign-top">awscli</td><td class="halign-right valign-top">121,743,694</td><td class="halign-right valign-top">0.83%</td></tr><tr><td class="halign-left valign-top">click</td><td class="halign-right valign-top">114,611,382</td><td class="halign-right valign-top">0.78%</td></tr><tr><td class="halign-left valign-top">wheel</td><td class="halign-right valign-top">112,656,886</td><td class="halign-right valign-top">0.76%</td></tr><tr><td class="halign-left valign-top">numpy</td><td class="halign-right valign-top">110,481,070</td><td class="halign-right valign-top">0.75%</td></tr><tr><td class="halign-left valign-top">cryptography</td><td class="halign-right valign-top">107,687,178</td><td class="halign-right valign-top">0.73%</td></tr><tr><td class="halign-left valign-top">rsa</td><td class="halign-right valign-top">101,669,487</td><td class="halign-right valign-top">0.69%</td></tr><tr><td class="halign-left valign-top">pyparsing</td><td class="halign-right valign-top">100,861,673</td><td class="halign-right valign-top">0.68%</td></tr></tbody></table></div>
<p>So, the 20 most downloaded packages account for 21% of all downloads. How far down do we have to go to account for 80%?</p>
<p>Well, the <a href="https://hugovk.github.io/top-pypi-packages/">top 5000 packages by download count are available here</a>, which you can total up like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="shell">➜ curl <span class="nt">-s</span> https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.json | jq .rows | jq <span class="nt">-r</span> <span class="s1">'(.[0] | keys_unsorted) as $keys | $keys, map([.[ $keys[] ]])[] | @csv'</span> | <span class="nb">cut</span> <span class="nt">-d</span><span class="s1">','</span> <span class="nt">-f1</span> | <span class="nb">awk</span> <span class="s1">'{total = total + $1}END{print total}'</span> | <span class="nb">numfmt</span> <span class="nt">--grouping</span>
13,225,311,500</code></pre></div>
<p>Ok, the top 5000 packages account for 13,225,311,500 downloads a month, so…​ 13,225,311,500 / 14,756,299,061 * 100 = <strong>89.62% of total downloads</strong> are accounted for by the top 5000 packages. In fact, the first 805 packages account for 80% of the downloads.</p>
<p>Perhaps unsurprisingly, if you plot that on a graph, it produces a perfect inverted power law curve, with a very long tail:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/supply-chain-attacks-package-managers/plot_pypi_downloads.svg" alt="Line graph" width="710px">
<figcaption>Figure 1. This tail continues to very slowly approach 100%, until you get down to the packages that have never been downloaded. The gray dotted lines show the 805th package accounting for 80% of the downloads.</figcaption></figure>
<p>So, you could put up a pypi.debian.org, with only 805 packages on and satisfy 80% of downloads - and only 5000 packages to satisfy 89% of downloads. Using our formula from above, you would need…​ only two maintainers for the 805 packages and only 13 maintainers for the 5000 package version. That sounds a lot more achievable!</p>
<p>These are almost certainly <em>also</em> the most actively updated packages, so you’d definitely need more maintainers than that - but even if you need 10 times that many, that’s still much more achievable.</p>
<p>But is that enough - and is it solving the right problem?</p></section>
<section class="doc-section level-1"><h2 id="_which_packages_are_the_problem">Which packages are the problem?</h2><p>Thinking about where supply chain attacks happen - it’s usually <em>not</em> the big packages. The most downloaded python package, <a href="https://github.com/boto/boto3">boto3</a>, is maintained by Amazon’s <span class="caps">AWS</span> team and has <em>many, many</em> eyeballs on it. It would be <em>extremely</em> hard to slip something malicious into boto.</p>
<figure class="image-block"><img src="https://imgs.xkcd.com/comics/dependency.png" alt="dependency">
<figcaption>Figure 2. That arrow is pointing to the ideal target for a supply chain attack: <a href="https://xkcd.com/2347/">xkcd #2347</a></figcaption></figure>
<p>I think this is <em>probably</em> the same for <em>most</em> of the popular packages - they have enough eyeballs on them already. The really juicy supply chain attacks are when you find some package that happens to be depended on by lots of other packages, but is developed <span class="amp">&</span> maintained by just one person. <a href="https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/">Leftpad</a> is the obvious example of this, but there are lots of others.</p>
<p>In my experience, most software project dependencies follow a power law too - they depend on a few big packages, and a larger number of smaller ones. If your package repository only covers the big packages, people will either have to fall back to PyPI for the little ones (leaving a supply chain attack hole), or more likely just continue to use PyPI for everything – defeating the purpose entirely.</p>
<p>Does this mean that you have to support <em>all the packages</em> to be useful? Possibly? If you <em>did</em> support all packages, that would certainly make it a no-brainer to switch and adoption would be much easier. But “just support all of PyPI” doesn’t seem like an achievable goal to me - I think you’d need some way to get started smaller and work your way up.</p>
<section class="doc-section level-2"><h3 id="_start_with_the_problem_packages">Start with the Problem Packages</h3><p>It seems to me that you could come up with a rough list of the problem packages - the ones that have few developers but lots of things depending on them - with only two pieces of data. You just need a list of all the packages on PyPI with: how many things depend on each, and how many developers work on them. It looks like the information you’d need is either <a href="https://console.cloud.google.com/bigquery?project=bigquery-public-data&page=table&t=downloads&d=pypi&p=bigquery-public-data"> available in Google’s BigQuery public datasets</a>, both for the PyPI <span class="amp">&</span> GitHub data, or in the dependency data from <a href="https://libraries.io/data">libraries.io</a>.</p>
<p>It seems to me that you could start with that list and maintain those packages in your vetted repo, and then just provide a transparent proxy to PyPI for the rest. You could then add to your list of verified packages over time, and anyone using your PyPI mirror would get less vulnerable to supply-chain attacks over time.</p></section></section>
<section class="doc-section level-1"><h2 id="_do_package_maintainers_actually_do_code_security_reviews">Do package maintainers actually do code <span class="amp">&</span> security reviews?</h2><p>I’m sure this varies <em>a lot</em> by package <span class="amp">&</span> maintainer, but I think the answer to this is mostly not, at least for Debian. They are involved in fixing bugs in the packages they maintain - but mostly bugs that affect packaging them up for Debian. I think they’re generally focussed on just the packaging part. That doesn’t mean that they couldn’t do code <span class="amp">&</span> security reviews, if that was the desired outcome.</p></section>
<section class="doc-section level-1"><h2 id="_i_think_people_would_pay_for_this">I think people would pay for this?</h2><p>As language package ecosystems grow, supply chain attacks seem to be on the rise, taking advantage of this new vector into the heart of organizational development teams.</p>
<p>Some of these organizations pay <a href="https://www.redhat.com/">Red Hat</a> a subscription for <a href="https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux">Red Hat Enterprise Linux</a>, which includes the <span class="caps">RPM</span> package repo’s - which provide this kind of service for the Fedora/Redhat package ecosystem. Some of these same organizations then get completely untrusted code directly from <span class="caps">NPM</span>/PyPI/Maven and just run it. It seems likely that some of them would probably <em>also</em> pay for something like pypi.redhat.com.</p>
<p>If you were paying developers full-time to maintain these packages, then <em>presumably</em> you could maintain more packages, with less people, than volunteer maintainers can in their spare time. This would further reduce the total number of maintainers you’d need.</p>
<hr>
<p>So, yeah, I think you could probably make it work, sustainably, without needing too many people. What do you think?</p>
<hr>
<section class="doc-section level-2"><h3 id="_references_footnotes">References <span class="amp">&</span> Footnotes</h3><div class="ulist"><ul><li><a href="https://drewdevault.com/2022/05/12/Supply-chain-when-will-we-learn.html">When will we learn? May 12, 2022,Drew DeVault</a></li><li><a href="https://nm.debian.org/public/people/dm_all/">Debian Maintainers List</a></li><li><a href="https://nm.debian.org/members/">Official members of the Debian project</a></li><li><a href="https://briancaffey.github.io/2017/12/02/arch-linux-package-data-analysis.html/">Analysis of <span class="caps">AUR</span> and Official Arch Repository data</a></li><li><a href="http://www.modulecounts.com/">Module Counts</a></li><li><a href="https://wiki.debian.org/Statistics">Debian Wiki: Statistics</a></li></ul></div></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">Numbers mostly from <a class="bare" href="http://www.modulecounts.com/">http://www.modulecounts.com/</a> <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">You could probably figure this out using data from <a class="bare" href="https://snapshot.debian.org/">https://snapshot.debian.org/</a> <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_3" role="doc-endnote">This is all configurable - apt <em>can</em> install packages for a single user, but doesn’t by default - but it can’t really install packages into a single folder. Similarly, pip <span class="amp">&</span> poetry <em>can</em> install packages globally, but that’s not the way most people use them. <a class="footnote-backref" href="#_footnoteref_3" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>Review: Sigur Rós World Tour, Vancouver 20222022-05-09T20:00:00-08:002022-05-09T20:00:00-08:00Duncan Locktag:duncanlock.net,2022-05-09:/blog/2022/05/09/review-sigur-ros-world-tour-vancouver-2022/<p class="lead">Good, but I didn’t feel very engaged with the music.</p>
<p>I was in the Lower Orchestra section, in the 16th row - and pretty tired, that Monday evening. I like their music, but I found myself drifting off, thinking about other things.</p>
<p>Thinking much at all is a bad sign for me when I’m watching live music. I usually find it very engrossing <span class="amp">&</span> uplifting - ideally, I get carried away by the music and exist fully in the moment for the duration. That didn’t happen for me at this gig. I felt rather disconnected from the performance - perhaps because it felt like it was happening far away?</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/review-sigur-ros-world-tour-vancouver-2022/PXL_20220510_034819638.MP.avif" alt="Photo from my seat, .">
<figcaption>Figure 1. The view from my seat</figcaption></figure>
<section class="doc-section level-1"><h2 id="_learnings">Learnings</h2><p>Things I learned from this gig:</p>
<div class="ulist"><ul><li>Only go and see people if you would listen to their stuff for an hour or two without …</li></ul></div></section><p class="lead">Good, but I didn’t feel very engaged with the music.</p>
<p>I was in the Lower Orchestra section, in the 16th row - and pretty tired, that Monday evening. I like their music, but I found myself drifting off, thinking about other things.</p>
<p>Thinking much at all is a bad sign for me when I’m watching live music. I usually find it very engrossing <span class="amp">&</span> uplifting - ideally, I get carried away by the music and exist fully in the moment for the duration. That didn’t happen for me at this gig. I felt rather disconnected from the performance - perhaps because it felt like it was happening far away?</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/review-sigur-ros-world-tour-vancouver-2022/PXL_20220510_034819638.MP.avif" alt="Photo from my seat, .">
<figcaption>Figure 1. The view from my seat</figcaption></figure>
<section class="doc-section level-1"><h2 id="_learnings">Learnings</h2><p>Things I learned from this gig:</p>
<div class="ulist"><ul><li>Only go and see people if you would listen to their stuff for an hour or two without a break, at home - because that’s what you will be doing at the gig.</li><li>Don’t sit down, 16 rows back, if you can help it. It’s <em>much</em> less engaging and exciting that way. At all-seater venues, try to be in the first 4 or 5 rows, if at all possible.</li><li>Sometimes it may be worth spending more to see it well - i.e. general admission or seat near the front - or don’t bother going at all.</li></ul></div></section>
<section class="doc-section level-1"><h2 id="_sigur_rós_world_tour_vancouver_2022"><a href="https://music.youtube.com/playlist?list=PLxMjPyxIGFZS9TiUYO7btRPZbmN-KZ5zv">Sigur Rós World Tour, Vancouver 2022</a></h2><p>May 9th at the Orpheum Theatre, Vancouver, <span class="caps">BC</span>, Canada</p>
<div class="table-block scrollable"><table class="frame-all grid-all stretch"><colgroup><col style="width: 37.5%;"><col style="width: 25%;"><col style="width: 25%;"><col style="width: 12.5%;"></colgroup><thead><tr><th class="halign-left valign-top">Title</th><th class="halign-left valign-top">Artist</th><th class="halign-left valign-top">Album</th><th class="halign-left valign-top">Duration</th></tr></thead><tbody><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/7ublWR5H0jONYPdLfPXtVaCarbnHrCM9YsBQLejVAVZ1DXEb-b0cS44aclA0sbTRjWDexWwR1ujDotgi=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Svefn-g-englar</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Ágætis byrjun</td><td class="halign-left valign-top">10:07</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/m3E-TJaf4LigGO5q-nEAo-BpMmlhO1zbgfpDS80bac-YVOiHkW-ElZZj-oE2oT99M-ifoM9INfS8Hvse=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Untitled #1 (Vaka)</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Untitled #1 (Vaka)</td><td class="halign-left valign-top">6:44</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/_VxvhcmdFJJUyEZjo52UWenUhDWGlEXImq9WFjW8G2CiOiOj8vBMbT6C9znIWqTYvBC7mcr-fTDH60Y=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Untitled #2 (Fyrsta)</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">( )</td><td class="halign-left valign-top">7:34</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/_VxvhcmdFJJUyEZjo52UWenUhDWGlEXImq9WFjW8G2CiOiOj8vBMbT6C9znIWqTYvBC7mcr-fTDH60Y=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Untitled #3 (Samskeyti)</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">( )</td><td class="halign-left valign-top">6:34</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/m1MkiLv3nO8ROJNZxxF-H7F3Ye1T6YT5WDj-Qca8n5vUCPTXRxQvLerRETYzOk5r7hxsFqE_mBLsfIjK=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Rafmagnið búið</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Ný batterí</td><td class="halign-left valign-top">4:53</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/7ublWR5H0jONYPdLfPXtVaCarbnHrCM9YsBQLejVAVZ1DXEb-b0cS44aclA0sbTRjWDexWwR1ujDotgi=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Ný batterí</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Ágætis byrjun</td><td class="halign-left valign-top">8:11</td></tr><tr><td class="halign-left valign-top"><img src="https://i.ytimg.com/vi/R6X6sNToF_U/sddefault.jpg?sqp=-oaymwEWCJADEOEBIAQqCghqEJQEGHgg6AJIWg&rs=AMzJL3nBOk61GsJVnWlR5ZqynydREYwwXA" alt="thumbnail" width="40" height="40" class="bare">“Gold 2” - Sigur Rós - Kings Theatre, Brooklyn, New York, 06.14.22</td><td class="halign-left valign-top">Lukretiah 101</td><td class="halign-left valign-top"></td><td class="halign-left valign-top">6:12</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/-D61B17z1wJrLbUzZuvm6GS1S7CwQSBh7JzOeF3vxuISzQS3qn_zdA2oCFadvTCfmifM0nQCw_Dz_lxj=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Fljótavík</td><td class="halign-left valign-top">12 Ensemble</td><td class="halign-left valign-top">Sigur Rós: Fljótavík</td><td class="halign-left valign-top">4:09</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/_VxvhcmdFJJUyEZjo52UWenUhDWGlEXImq9WFjW8G2CiOiOj8vBMbT6C9znIWqTYvBC7mcr-fTDH60Y=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Untitled #7 (Dauðalagið)</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">( )</td><td class="halign-left valign-top">13:00</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/m3E-TJaf4LigGO5q-nEAo-BpMmlhO1zbgfpDS80bac-YVOiHkW-ElZZj-oE2oT99M-ifoM9INfS8Hvse=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Untitled #9 (Smáskfia 1)</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Untitled #1 (Vaka)</td><td class="halign-left valign-top">4:39</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kd3bVCgH8gf2OiqOgoeRWFKknL12YMnLxzvKM4fMrqwcrHWFNZvEL8oY3_41MpcCC0jJt8hJCIVhxpc5mw=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Glósóli</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Takk…​</td><td class="halign-left valign-top">6:15</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/_VxvhcmdFJJUyEZjo52UWenUhDWGlEXImq9WFjW8G2CiOiOj8vBMbT6C9znIWqTYvBC7mcr-fTDH60Y=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Untitled #6 (E-Bow)</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">( )</td><td class="halign-left valign-top">8:49</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/_qhgsSvWVndPc7Ix5r_L2drzBVElOGRLdASrzloz-hrjxx0i7EGxmb03B4tB-CZlw7wz0S2Ta-mqLzlF=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Ekki múkk</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Valtari</td><td class="halign-left valign-top">7:46</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kd3bVCgH8gf2OiqOgoeRWFKknL12YMnLxzvKM4fMrqwcrHWFNZvEL8oY3_41MpcCC0jJt8hJCIVhxpc5mw=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Sæglópur</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Takk…​</td><td class="halign-left valign-top">7:38</td></tr><tr><td class="halign-left valign-top"><img src="https://i.ytimg.com/vi/KxqM2fIuph0/sddefault.jpg?sqp=-oaymwEWCJADEOEBIAQqCghqEJQEGHgg6AJIWg&rs=AMzJL3nEgT_SXhqDFY017K3qPxi0-wC0VQ" alt="thumbnail" width="40" height="40" class="bare">“Gold 4” - Sigur Rós - Kings Theatre, Brooklyn, New York, 06.14.22</td><td class="halign-left valign-top">Lukretiah 101</td><td class="halign-left valign-top"></td><td class="halign-left valign-top">12:03</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/lFk-FN3gEGtaEPx3-a_1nHJkTClW5OO59_2-3gU_mIOvx_9SnYoOz1TkfmAHXrtSv7eWb9scnF6e6KGF=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Festival</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Með suð í eyrum við spilum endalaust</td><td class="halign-left valign-top">9:26</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/v3DPaA7k1HSOO-xzdcWpF8OG_wUDezGcA68W3y7yjBx6f97fYkDN1M8PmyhMrEJV5ziGP12pZ_X6yR2H=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Kveikur</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">Kveikur</td><td class="halign-left valign-top">5:56</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/_VxvhcmdFJJUyEZjo52UWenUhDWGlEXImq9WFjW8G2CiOiOj8vBMbT6C9znIWqTYvBC7mcr-fTDH60Y=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Untitled #8 (Popplagið)</td><td class="halign-left valign-top">Sigur Rós</td><td class="halign-left valign-top">( )</td><td class="halign-left valign-top">11:46</td></tr></tbody></table></div></section>Using Windows after 15 years on Linux2022-04-06T16:57:38-07:002022-04-06T16:57:38-07:00Duncan Locktag:duncanlock.net,2022-04-06:/blog/2022/04/06/using-windows-after-15-years-on-linux/<p class="lead">I’ve been using Linux exclusively for ~15 yrs.
I’ve recently started a fantastic new job – the only wrinkle was that it came with a Windows 10 laptop.
This is my first time using Windows after a 15-year break.
This is how it’s been going.</p>
<section class="doc-section level-1"><h2 id="_first_impressions">First Impressions</h2><p>Windows is <em>such a mess</em>! It’s sort of shocking how much of a mess it is. Desktop Linux is often criticized for this, but Windows is much worse, somehow! It’s <em>really</em> inconsistent. Half of it is “new” <span class="caps">UI</span> and half of it is old Win32/<span class="caps">GDI</span> type <span class="caps">UI</span> - just as bad as <span class="caps">KDE</span>/<span class="caps">GTK</span> - except <em>worse</em>, because you can’t configure them to use the same theme. Also, when you install a Linux distribution, it’ll start off either all <span class="caps">KDE</span> or all <span class="caps">GTK</span>, or whatever - but with Windows …</p></section><p class="lead">I’ve been using Linux exclusively for ~15 yrs.
I’ve recently started a fantastic new job – the only wrinkle was that it came with a Windows 10 laptop.
This is my first time using Windows after a 15-year break.
This is how it’s been going.</p>
<section class="doc-section level-1"><h2 id="_first_impressions">First Impressions</h2><p>Windows is <em>such a mess</em>! It’s sort of shocking how much of a mess it is. Desktop Linux is often criticized for this, but Windows is much worse, somehow! It’s <em>really</em> inconsistent. Half of it is “new” <span class="caps">UI</span> and half of it is old Win32/<span class="caps">GDI</span> type <span class="caps">UI</span> - just as bad as <span class="caps">KDE</span>/<span class="caps">GTK</span> - except <em>worse</em>, because you can’t configure them to use the same theme. Also, when you install a Linux distribution, it’ll start off either all <span class="caps">KDE</span> or all <span class="caps">GTK</span>, or whatever - but with Windows you’re stuck with a random mix of both <em>right from the start</em>.</p>
<p>Thankfully, there <em>is</em> a dark theme available - but <em>only</em> for “new” <span class="caps">UI</span> things, naturally. So as soon as you randomly stumble into some old non-themed <span class="caps">UI</span>, suddenly <span class="caps">BRIGHT</span> <span class="caps">WHITE</span>!</p></section>
<section class="doc-section level-1"><h2 id="_you_cant_customize_anything">You can’t customize anything!</h2><p>You’re pretty much stuck with whatever Microsoft decides to give you, compared to Linux - which is <em>completely</em> customizable: you can build your own version of Linux from the ground up, choosing every software building block yourself, if you choose to. Like almost all Linux users, I <em>don’t</em> choose to - I use a Linux distribution (or “distro”), where someone else has made these choices for me. This lets me pick the distro that’s closest to my needs and customize anything I want to change. I’ve been using <a href="https://xubuntu.org/">Xubuntu</a> for years and it suits me - but there are <a href="https://distrowatch.com/">hundreds of Linux Distro’s to choose</a> from.</p></section>
<section class="doc-section level-1"><h2 id="_windows_wasnt_made_for_me">Windows Wasn’t Made for Me</h2><p>I am a software <span class="amp">&</span> web developer - and Linux is a toolbox, full of highly polished tools, crafted over decades by software developers, for software developers. Windows is…​ not that. It’s a commercial <span class="caps">OS</span>, aimed at users of Word, Excel <span class="amp">&</span> Outlook, pretty much. You can <em>feel</em> this difference all the time that you’re using it - it pervades everything.</p>
<section class="doc-section level-2"><h3 id="_non_composable_software">Non-composable Software</h3><p>The command line tools (echo, cat, grep, sed, awk, find, cut, sort, curl, ssh, etc…​) which make up the standard Linux/Unix toolbox are all composable and general purpose. You can join them together like Lego bricks, in whatever combination you like, to make new tools on the fly. You do this on the command line, by piping streams of text from one tool to another and using them to transform it however you need.</p>
<p>Sadly, nobody has ever <em>really</em> figured out how to make <span class="caps">GUI</span> software like this - general purpose <span class="amp">&</span> composable. Windows has always focussed heavily on the <span class="caps">GUI</span>, to the almost complete exclusion of the command line - which means that it doesn’t have this foundation of composable software tools. Almost everything is a special purpose piece of <span class="caps">GUI</span> software. Which you have to go and find. And then download and install.</p>
<section class="doc-section level-3"><h4 id="_solution">Solution:</h4><p>You can get Windows versions of most of the <a href="https://en.wikipedia.org/wiki/List_of_GNU_Core_Utilities_commands">standard *nix userland utils</a>, which seem to work <span class="caps">OK</span> with PowerShell:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>scoop <span class="nb">install </span>coreutils</code></pre></div>
<p>(See <a href="#_installing_software">installing software for more on package managers <span class="amp">&</span> scoop.</a>)</p></section></section>
<section class="doc-section level-2"><h3 id="_paths">Paths</h3><p>Stupidly long paths with lots of spaces in don’t matter if you’re only ever clicking on things in the <span class="caps">GUI</span> file manager - but if you’re trying to use the command line, then they’re just constant friction.</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 50%;"><col style="width: 50%;"></colgroup><thead><tr><th class="halign-left valign-top">Windows Path</th><th class="halign-left valign-top">Linux Path</th></tr></thead><tbody><tr><td class="halign-left valign-top">C:\Users\DuncanLock\</td><td class="halign-left valign-top">/home/duncan or just ~</td></tr><tr><td class="halign-left valign-top">C:\Users\DuncanLock\Documents\PowerShell\Microsoft.PowerShell_profile.ps1</td><td class="halign-left valign-top">~/.bashrc</td></tr><tr><td class="halign-left valign-top">C:\Users\DuncanLock\AppData\Local</td><td class="halign-left valign-top">~/usr/bin or ~/bin</td></tr></tbody></table></div>
<p>…​and on and on. In fairness, they <em>have</em> got rid of a lot of the spaces in the paths since I last used Windows.</p>
<p>Thankfully, PowerShell does have a <code>~</code> alias for your home folder, and <code>cd ~</code> works!</p></section>
<section class="doc-section level-2"><h3 id="_environment_variables">Environment Variables</h3><p>Environment Variables look like this: <code>%PROFILE%</code>, instead of this: <code>$HOME</code>; which is fine, just different; although PowerShell seems to accept either form, which is nice.</p>
<p>What <em>isn’t fine</em>, is that they’re stored in the Windows Registry, apparently? If you update these, either using <code>setx %var% value</code>, or the <span class="caps">GUI</span>…​ <em>you have to restart your shell</em> - i.e. you type <code>exit</code>, the window/tab goes away, and you open another one. Just let that sink in for a moment.</p>
<section class="doc-section level-3"><h4 id="_solution_2">Solution:</h4><p>I haven’t tried this, but I think if you want to add persistent environment variables to your currently running shell, you should put a <code>setx</code> command in your <code>$profile</code> file and then reload it: <code>. $profile</code> - or maybe run <code>myvar="value" && setx %myvar% "value"</code>, or something similar.</p></section></section></section>
<section class="doc-section level-1"><h2 id="_installing_software">Installing Software</h2><p>Installing things is still, mostly, going to random websites, downloading an <code>.exe</code> (or a <code>.msi</code> if you’re lucky) and running it! Holy shit! You now get some click through warnings when you try to run them, but it’s still the only way to install lots of things! This is slightly terrifying and pretty mind-blowing in 2022!</p>
<p>There <em>is</em> the Microsoft Store now, but it’s just…​ awful? It manages to be both a ghost town and a cesspool of scam/shovelware at the same time, somehow? It’s got less useful stuff in it than most Linux distro “app stores” and is <em>utterly miniscule</em> compared to the Debian repositories, which have ~60,000 packages in, or Arch’s <span class="caps">AUR</span>, with 73,000 (these counts include the whole Linux <span class="caps">OS</span>, though, with is installed using the same package manager).</p>
<p>Like Netflix, the <span class="caps">MS</span> Store deliberately makes it hard to see exactly how much/little stuff they have, but in most useful categories, it’s fairly empty. It <em>usually</em> doesn’t have what I need, so far.</p>
<aside class="sidebar"><p>For example, trying to install a font? If you’ve already downloaded it you can right-click on it - or the new Font Install thing in settings will install local font files for you; but it will <em>also</em> suggest you visit the <span class="caps">MS</span> Store - where there are a grand total of 19 fonts available!?</p>
<p>A cursory <code>apt list | rg 'ttf|otf' | wc -l</code> says there are 101 in the repositories on my Ubuntu box. Or Google fonts, which currently has 1364 font families. These are all open source, freely licensed and the fonts work fine on Windows - Microsoft could just import all these into the store, if they felt like it.</p></aside>
<p><span class="caps">MS</span> <em>could</em> have fixed this with their Store, if that had a proper package manager underneath - but they don’t seem to have done this. Some team at <span class="caps">MS</span> decided to rip-off AppGet (<a href="https://keivan.io/the-day-appget-died/">killing it</a>) to create WinGet and then <a href="https://niemarwinget.medium.com/winget-is-terrible-i-want-appget-back-41b3ca598596">mostly abandon it</a>. Some other team created <span class="caps">MSIX</span>. Some other team created the Store. Etc…​</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/using-windows-after-15-years-on-linux/ms_organizational_chart.webp" alt="Diagram of an imagined MS Org Chart. Each team is in a sealed bubble" width="pointing guns at each other" height="only linked to the top of the org">
<figcaption>Figure 1. Microsoft Org Chart, from <a class="bare" href="https://bonkersworld.net/organizational-charts">https://bonkersworld.net/organizational-charts</a>. This explains a lot of things. I’m not sure what the open-source version of this looks like, but somehow they made package managers work at scale.</figcaption></figure>
<p>Some “package managers” have appeared for Windows recently - nuget, choclatey, scoop, winget, etc…​ These are just papering over the cracks. They don’t have their own package repositories, reproducible builds, package signing, etc…​ - because lots of things in Windows aren’t open source, so they mostly can’t. They just automate the process of going to the website, downloading an installer and then running it - which <em>is</em> slightly better than doing it yourself.</p>
<p>This isn’t really proper package management - it’s just automated download <span class="amp">&</span> install. Linux package managers do the following things:</p>
<div class="ulist"><ul><li>Have a package repository of some kind, mostly storing package lists <span class="amp">&</span> metadata, binary installation packages, but sometimes source code</li><li>Package signing, reproducible builds, secure downloads, etc…​</li><li>Browse <span class="amp">&</span> search these packages</li><li>Install a package from the repository locally. This means:<ul><li>If it’s a source package system, compile the source and test it</li><li>Copy the files to the system, put them in the right places, and run optional installation scripts and hooks.</li><li>Record all files that are installed and know which file belongs to which package.</li></ul></li><li>When uninstalling:<ul><li>Remove all the packages files automatically</li><li>Run optional uninstallation scripts and hooks.</li><li>Make sure the software is uninstalled safely and completely.</li></ul></li><li>Be able to update packages. This means:<ul><li>Knowing what packages are installed, and what version</li><li>Having the ability to download <span class="amp">&</span> update any outdated packages.</li></ul></li></ul></div>
<p>Importantly, the whole <span class="caps">OS</span> - and all applications - are installed through this system. Everything is installed the same way - and updated the same way.</p>
<aside class="sidebar"><p>This is to say nothing of the entirely next-level stuff going on with the <a href="https://en.wikipedia.org/wiki/Nix_package_manager">Nix Package Manager</a> and the <a href="https://nixos.org/explore.html">NixOS</a></p></aside>
<p>To be fair to <span class="caps">MS</span>, the <a href="https://en.wikipedia.org/wiki/Windows_Installer"><span class="caps">MSI</span> system</a> <em>does</em> do <em>some</em> of this - but <span class="caps">MSI</span> packages have been a pain to create since the start, and apparently still are. So a lot of software <em>still</em> doesn’t use <span class="caps">MSI</span> files, 20 years later - they still come with a Nullsoft/ InstallShield/ Inno/ homegrown setup.exe. To be even fairer, there is <a href="https://docs.microsoft.com/en-us/windows/msix/overview">a new <span class="caps">MSIX</span> system</a>, which adds containerization/sandboxing of apps (like flatpak/snap) as well as updating - while being simpler to create - but it’s new…​ so almost no-one uses it.</p>
<p>Even taking that into account, there doesn’t seem to be a notion that the system as a whole could provide some kind of cohesive application install <span class="amp">&</span> update experience. Windows Update <em>almost</em> provides this for the <span class="caps">OS</span>, but applications are all updated separately, even if they’re using the new <span class="caps">MSIX</span> installer system, afaik. I think the <span class="caps">MS</span> Store <em>is supposed to do this</em>, for the tiny number of apps that you can install from there? Maybe this is just me coming in while this is all party-way through being fixed?</p>
<p>In addition to this, these various “package managers” don’t all have the same packages/apps available, so you will probably end up with several of them installed eventually - and have to remember which thing you installed something with when you want to update it - if that “package manager” even supports updating, because not all of them do!</p>
<p>I fully understand the historical <span class="amp">&</span> commercial reasons <em>why</em> it’s like this, but it’s still a pretty poor experience for the end user, compared to Linux.</p>
<section class="doc-section level-2"><h3 id="_solution_3">Solution:</h3><p><a href="https://scoop.sh/">Scoop</a> seems to be the best of the bunch, so far?</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>scoop search wget
<span class="gp">$</span><span class="w"> </span>scoop info postgres
<span class="gp">$</span><span class="w"> </span>scoop <span class="nb">install </span>coreutils wget xh bat ripgrep</code></pre></div></section></section>
<section class="doc-section level-1"><h2 id="_a_terminal_that_doesnt_suck">A Terminal That Doesn’t Suck</h2><p>The shells <span class="amp">&</span> terminal applications that come installed on Windows are just astonishingly bad. Like 1980s <span class="caps">DOS</span> in a window bad - and they haven’t changed <em>at all</em> since I last used them ~15 yrs ago. Just head shakingly awful. There’s a new <em>shell</em> now, called <a href="https://docs.microsoft.com/en-us/powershell/scripting/overview">PowerShell</a>, to run <em>inside</em> your terrible 1980s terminal window, but the terminal it’s running inside, still sucks.</p>
<p>I sometimes wondered why the <span class="caps">VS</span> Code team put so much effort into the built-in terminal inside the editor. I tried it once on Linux and never touched it again, because the terminal window I had right next to my editor was just massively better in every way. Having used Windows terminals for a while, I now <em>fully understand why it’s there</em>.</p>
<section class="doc-section level-2"><h3 id="_solution_4">Solution:</h3><p>It turns out that answer to this is to install <a href="https://www.microsoft.com/en-US/p/windows-terminal/9n0dx20hk701?activetab=pivot:overviewtab">Windows Terminal</a>. It’s not <a href="https://sw.kovidgoyal.net/kitty/">Kitty</a>, but at least it doesn’t suck.</p>
<p>The shell that’s running <em>inside</em> this Terminal is configurable, but I’m currently using PowerShell. It’s quite powerful, if you’re willing to learn it, I think. It also comes with enough aliases for things like <code>ls → dir</code> that it’s fairly comfortable coming from Bash<a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>. It’s certainly a massive improvement over cmd.exe + whatever the old shell was called.</p></section></section>
<section class="doc-section level-1"><h2 id="_did_it_just_restart_itself_and_lose_all_my_terminals">Did it Just…​ Restart Itself and Lose All My Terminals!?</h2><p>I was surprised by this one morning, when I came back to my Windows machine, it had lost all my open terminals <span class="amp">&</span> <span class="caps">SSH</span> sessions overnight, as well as all my VSCode windows. My Outlook, Teams and Edge windows were all still there, so <span class="caps">WFT</span>!? I initially thought they’d crashed, but after trawling through Event Viewer, I discovered that Windows Update had decided to restart the machine without asking me!</p>
<p>Screw you software, I’m in charge, not you.</p>
<p>Turns out that some Windows Apps are “Restartable” and some aren’t - which means they get reloaded with all their windows when you restart.</p>
<p>I eventually found that Windows Terminal Preview Edition is now “Restartable” too. Sadly, but unsurprisingly, this just means that your terminal windows <span class="amp">&</span> tabs come back, but not their contents, or <span class="caps">SSH</span> sessions, etc…​</p>
<p>Update: It did it again! But the Lenovo thing that’s been nagging me to install a <span class="caps">BIOS</span> update, which unsurprisingly also requires a restart, is <em>still</em> nagging me. Because there’s no system-wide package manager, so all these little things have their own installers and don’t co-ordinate anything. Ugh.</p>
<p>This is not how we do things in Linux land:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">uptime</span>
<span class="go">09:33:15 up 56 days, 16:33, 1 user, load average: 1.36, 1.29, 0.91</span></code></pre></div>
<section class="doc-section level-2"><h3 id="_solution_5">Solution:</h3><p>You <em>can</em> <a href="https://duckduckgo.com/?q=windows+update+disable+restart">turn this off</a> - unless your <span class="caps">IT</span> dept has set this by policy, which is the case for me.</p></section></section>
<section class="doc-section level-1"><h2 id="_virtual_desktops_workspaces">Virtual Desktops / Workspaces</h2><p>Linux has had rock-solid multiple virtual desktop/workspaces support forever (30 yrs?) - Windows <em>just</em> got this in Windows 10. It was <em>possible</em> before via hacky 3rd party software, but it was <em>very</em> hacky and didn’t work very well, in my experience.</p>
<p>Using multiple workspaces/virtual desktops is a core part of my workflow on Linux - I currently have 20 of them, so this is fairly important to me.</p>
<p>The one in Windows 10 seems to work <em>better</em> than the previous hacky 3rd party ones, although I have some issues with it:</p>
<div class="ulist"><ul><li>There’s no way to see which desktop you’re on, except by going to the switcher.</li><li>The win+tab desktop switcher screen does a lot of things - including the only way to move windows between desktops, afaik. It can be a bit slow/janky.</li><li>When apps get restarted, <a href="https://answers.microsoft.com/en-us/windows/forum/all/restart-apps-to-the-right-desktop/7d534448-fd8e-4a62-ada8-50799e837826">they all end up on the first desktop</a>, because, clearly, people at <span class="caps">MS</span> don’t use multiple desktops.</li></ul></div>
<section class="doc-section level-2"><h3 id="_solutions">Solutions:</h3><div class="ulist"><ul><li>Install <a href="https://github.com/zgdump/windows-virtualdesktopindicator">Virtual Desktop Indicator</a> to get a desktop number indicator in your taskbar, popup desktop name on switch and mouse wheel switching.</li><li>Install <a href="https://github.com/kangyu-california/PersistentWindows">PersistentWindows</a> - which keeps track of window positions in real time, and automatically restores window layout to last matching monitor setup.</li></ul></div></section></section>
<section class="doc-section level-1"><h2 id="_windows_all_moved_to_main_monitor_after_sleepoff">Windows all Moved to Main Monitor After Sleep/Off</h2><p>It seems that when Windows sends DisplayPort monitors to sleep, it immediately forgets that it did this, and acts like you disconnected them - and moves all your Windows that used to be on these monitors onto the primary monitor. Every. Single. Time.
So, when you wake it up, you have to put all your windows back where they were. Every. Single. Time.</p>
<p>This seems to have been happening since Windows 7, with hundreds of people complaining, to no avail:</p>
<div class="ulist"><ul><li><a class="bare" href="https://answers.microsoft.com/en-us/windows/forum/all/active-windows-all-moved-to-main-monitor-after/42396920-908c-486f-800b-ff4035337b35">https://answers.microsoft.com/en-us/windows/forum/all/active-windows-all-moved-to-main-monitor-after/42396920-908c-486f-800b-ff4035337b35</a></li><li><a class="bare" href="https://answers.microsoft.com/en-us/windows/forum/all/windows-10-multiple-display-windows-are-moved-and/2b9d5a18-45cc-4c50-b16e-fd95dbf27ff3">https://answers.microsoft.com/en-us/windows/forum/all/windows-10-multiple-display-windows-are-moved-and/2b9d5a18-45cc-4c50-b16e-fd95dbf27ff3</a></li><li><a class="bare" href="https://answers.microsoft.com/en-us/windows/forum/windows_7-hardware/windows-7-movesresizes-windows-on-monitor-power/1653aafb-848b-464a-8c69-1a68fbd106aa?page=8&tm=1439182229675">https://answers.microsoft.com/en-us/windows/forum/windows_7-hardware/windows-7-movesresizes-windows-on-monitor-power/1653aafb-848b-464a-8c69-1a68fbd106aa?page=8&tm=1439182229675</a></li></ul></div>
<section class="doc-section level-2"><h3 id="_solutions_2">Solutions:</h3><div class="ulist"><ul><li>Set screen sleep timeouts really long, or off when plugged in</li><li>Install <a href="https://github.com/kangyu-california/PersistentWindows">PersistentWindows</a> - which keeps track of window positions in real time, and automatically restores window layout to last matching monitor setup.</li></ul></div></section></section>
<section class="doc-section level-1"><h2 id="_no_middle_click_paste">No Middle-Click Paste</h2><p>Linux (well, really the window managers, so X11 and then, i3, Wayland, etc…​) have multiple clipboards. The ones I care about are the Primary selection one, and the Secondary one. The names are historical accidents, but the “primary” one always has a copy of the last text you selected from anywhere, which can be pasted anywhere by clicking the middle mouse button. You just select some text and that’s it - you don’t have to do anything else and you can then middle-click paste this anywhere. The “secondary” clipboard is the “normal” Cut, Copy, Paste, Ctrl+c, Ctrl+v one.</p>
<p>To be clear - I don’t really care about multiple clipboards or history or a clipboard manager, I just want the Primary Selection <span class="amp">&</span> middle-click paste thing.</p>
<p>I just want any text selection anywhere to be automatically kept somewhere and allow me to paste it anywhere on a middle mouse click. This is <em>incredibly</em> useful and Windows just doesn’t have anything equivalent at all.</p>
<p>Windows Terminal <em>will</em> paste the contents of the clipboard on a right click, <em>and</em> does copy the last selection to the clipboard - which is <em>close</em> - but this doesn’t work anywhere else, sadly, only in Windows Terminal; I want this everywhere. Also, because there’s only <em>one</em> clipboard, every time you select <em>anything</em> in Windows Terminal, it overwrites the clipboard - which is why Linux has another clipboard, just for this.</p>
<section class="doc-section level-2"><h3 id="_solution_6">Solution:</h3><p>This originally said “None?”, but I got some great suggestion in the comments and over on <a href="https://news.ycombinator.com/item?id=30944438"><span class="caps">HN</span></a>, so there are some options now:</p>
<div class="ulist"><ul><li><a href="https://www.highrez.co.uk/downloads/XMouseButtonControl.htm">X-Mouse Button Control</a></li><li><a href="https://www.autohotkey.com/">AutoHotkey</a> and <a href="https://www.autohotkey.com/board/topic/5139-auto-copy-selected-text-to-clipboard">some scripting</a></li><li><a href="http://fy.chalmers.se/~appro/nt/TXMouse/">TXMouse</a>, maybe, although no idea if it still works in Windows 10 since it hasn’t been updated since 2005</li></ul></div></section></section>
<section class="doc-section level-1"><h2 id="_ssh"><span class="caps">SSH</span></h2><p>I tried PuTTY, which I’d heard was <em>the</em> good <span class="caps">SSH</span> thing on Windows, but it’s…​ not good, at all.
<a href="https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse">PowerShell does come with an <span class="caps">SSH</span> client</a>, so once you have this working with a reasonable terminal, you can use <span class="caps">SSH</span> as normal.</p>
<section class="doc-section level-2"><h3 id="_solution_7">Solution:</h3><div class="ulist"><ul><li>Use the <a href="https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse"><span class="caps">SSH</span> that comes with Windows/PowerShell</a></li></ul></div></section></section>
<section class="doc-section level-1"><h2 id="_keyboard_shortcuts">Keyboard Shortcuts</h2><p>A few useful keyboard shortcuts I’ve started using:</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 50%;"><col style="width: 50%;"></colgroup><thead><tr><th class="halign-left valign-top">Key Combo</th><th class="halign-left valign-top">Function</th></tr></thead><tbody><tr><td class="halign-left valign-top">Win+Ctrl+left/right arrow</td><td class="halign-left valign-top">Switch virtual desktop left/right</td></tr><tr><td class="halign-left valign-top">Win+Tab</td><td class="halign-left valign-top">Open the desktop switcher screen.</td></tr><tr><td class="halign-left valign-top">Win+e</td><td class="halign-left valign-top">Open the file manager</td></tr><tr><td class="halign-left valign-top">Win+x</td><td class="halign-left valign-top">Power users menu</td></tr></tbody></table></div>
<p>Again, these aren’t very customizable - you can get <em>some</em> more customization of global hotkeys by using the <a href="https://github.com/microsoft/PowerToys">PowerToys</a> <a href="https://docs.microsoft.com/en-gb/windows/powertoys/keyboard-manager">Keyboard Manager</a> thing, but there are lots of combination that are seemingly just not possible, for some reason; I’d like to use <code>Win+Enter</code> to launch a new terminal window, to match my Linux workflow, but that combination isn’t allowed.</p></section>
<section class="doc-section level-1"><h2 id="_things_that_are_better_on_windows">Things That are Better on Windows</h2><div class="ulist"><ul><li>Firmware installation as part of Windows Update seems to just work, so far.</li><li>…​ that’s it?</li></ul></div>
<hr></section>
<section class="doc-section level-1"><h2 id="_footnotes_references">Footnotes <span class="amp">&</span> References</h2></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">To see a list of all the currently defined aliases, run <code>Get-Alias</code>. <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>Write to Minister Guilbeault Opposing Bay Du Nord Offshore Oil Development2022-03-20T09:04:45-07:002022-03-20T09:04:45-07:00Duncan Locktag:duncanlock.net,2022-03-20:/blog/2022/03/20/write-to-minister-guilbeault-opposing-bay-du-nord-offshore-oil-development/<p>The last thing we need right now is new fossil fuel projects. Send a message to Minister Steven Guilbeault and key cabinet ministers, calling on them to reject the Bay Du Nord project and investing in a just transition instead: <a class="bare" href="https://act.leadnow.ca/bay-du-nord-ett/">https://act.leadnow.ca/bay-du-nord-ett/</a></p>
<p>This is what I wrote, if anyone wants some ideas:</p>
<div class="quote-block"><blockquote><p>To Minister Guilbeault and cabinet,</p>
<p>I’m writing to you to urge you to reject the Bay du Nord offshore drilling project.</p>
<p>Oil and Gas is ~26% of Canada’s emissions - <span class="caps">AND</span> this is <span class="caps">ONLY</span> the emissions from production - not the emissions from burning all that fuel for transport, heating etc…​ the Oil and Gas industry needs to go away, <span class="caps">ASAP</span>. Fossil fuels must stay in the ground to have any chance of getting to zero.</p>
<p>No approvals for new fossil fuel extraction projects, ever …</p></blockquote></div><p>The last thing we need right now is new fossil fuel projects. Send a message to Minister Steven Guilbeault and key cabinet ministers, calling on them to reject the Bay Du Nord project and investing in a just transition instead: <a class="bare" href="https://act.leadnow.ca/bay-du-nord-ett/">https://act.leadnow.ca/bay-du-nord-ett/</a></p>
<p>This is what I wrote, if anyone wants some ideas:</p>
<div class="quote-block"><blockquote><p>To Minister Guilbeault and cabinet,</p>
<p>I’m writing to you to urge you to reject the Bay du Nord offshore drilling project.</p>
<p>Oil and Gas is ~26% of Canada’s emissions - <span class="caps">AND</span> this is <span class="caps">ONLY</span> the emissions from production - not the emissions from burning all that fuel for transport, heating etc…​ the Oil and Gas industry needs to go away, <span class="caps">ASAP</span>. Fossil fuels must stay in the ground to have any chance of getting to zero.</p>
<p>No approvals for new fossil fuel extraction projects, ever.</p>
<p>This project would move us further from our climate targets and lock us into years of climate destruction. In addition, its location would put critical marine biodiversity at risk.</p>
<p>The science is clear: to change course on climate and build safe, thriving communities, our reliance on fossil fuels must end and we must transition to a sustainable economy. Your government must recognize this and invest in sustainable jobs and energy sources as part of a just transition away from fossil fuels.</p>
<p>Sincerely,</p>
<p>Duncan Lock
<span class="caps">V5L</span> 2W4</p></blockquote></div>Review: Nick Cave & Warren Ellis Gig, Vancouver 20222022-03-16T19:30:00-07:002023-05-24T09:48:21-07:00Duncan Locktag:duncanlock.net,2022-03-16:/blog/2022/03/16/review-nick-cave-and-warren-ellis-gig-vancouver-2022/<p class="lead">A spellbinding <span class="amp">&</span> breathtaking evening, with a couple of surprising jewels, in a night full of riches.</p>
<p>Just got back from seeing Nick Cave and Warren Ellis in Vancouver. What a joy and a privilege! I’ve missed live music so much - what an incredible night!</p>
<p>This is a video from that night in Vancouver, of their cover of Cosmic Dancer, which was a fantastic surprise for me. That video was taken by someone standing just in front of me!</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/d0cf1DRmhcY?rel=0" frameborder="0" allowfullscreen></iframe></div>
<p>We got <em>two</em> encores. The first one was Hollywood, Henry Lee <span class="amp">&</span> Girl in Amber - all great. Then they cleared the stage and lots of people left. Then we got a second encore!</p>
<p>This is Into My Arms, which he did with just the amazing backing singers, as a second encore, after half the audience had left thinking it was done - one …</p><p class="lead">A spellbinding <span class="amp">&</span> breathtaking evening, with a couple of surprising jewels, in a night full of riches.</p>
<p>Just got back from seeing Nick Cave and Warren Ellis in Vancouver. What a joy and a privilege! I’ve missed live music so much - what an incredible night!</p>
<p>This is a video from that night in Vancouver, of their cover of Cosmic Dancer, which was a fantastic surprise for me. That video was taken by someone standing just in front of me!</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/d0cf1DRmhcY?rel=0" frameborder="0" allowfullscreen></iframe></div>
<p>We got <em>two</em> encores. The first one was Hollywood, Henry Lee <span class="amp">&</span> Girl in Amber - all great. Then they cleared the stage and lots of people left. Then we got a second encore!</p>
<p>This is Into My Arms, which he did with just the amazing backing singers, as a second encore, after half the audience had left thinking it was done - one of several completely spellbinding moments:</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/16Dy3T1h8AI?rel=0" frameborder="0" allowfullscreen></iframe></div>
<section class="doc-section level-1"><h2 id="_learnings">Learnings</h2><p>Things I learned from this gig:</p>
<div class="ulist"><ul><li>This <em>really</em> re-enforced my earlier lessons from the <a href="https://duncanlock.net/blog/2022/05/09/review-sigur-ros-world-tour-vancouver-2022/">Sigur Rós gig</a> - be at the front <span class="amp">&</span> only go see people you would happily sit down and listen to for a solid hour at home.</li><li>If you’re enjoying it, stay until the very end; if you can, make sure you don’t have to be somewhere afterwards.</li></ul></div></section>
<section class="doc-section level-1"><h2 id="_nick_cave_warren_ellis_tour_setlist_vancouver_2022"><a href="https://music.youtube.com/playlist?list=PLxMjPyxIGFZSJeKkzppWiDQKzl2dLYuYn">Nick Cave <span class="amp">&</span> Warren Ellis Tour Setlist, Vancouver 2022</a></h2><p>March 16th, at the Queen Elizabeth Theatre</p>
<div class="table-block scrollable"><table class="frame-all grid-all stretch"><colgroup><col style="width: 37.5%;"><col style="width: 25%;"><col style="width: 25%;"><col style="width: 12.5%;"></colgroup><thead><tr><th class="halign-left valign-top">Title</th><th class="halign-left valign-top">Artist</th><th class="halign-left valign-top">Album</th><th class="halign-left valign-top">Duration</th></tr></thead><tbody><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/tdG06VqUIgn7UB0NxtoIcTEjkRiD9v9zfvmWITk309_jCYlKOkuMAEJUnDWhk-SydK1fJxA085PKUTy45g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Spinning Song</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Ghosteen</td><td class="halign-left valign-top">4:44</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/tdG06VqUIgn7UB0NxtoIcTEjkRiD9v9zfvmWITk309_jCYlKOkuMAEJUnDWhk-SydK1fJxA085PKUTy45g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Bright Horses</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Ghosteen</td><td class="halign-left valign-top">4:53</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/tdG06VqUIgn7UB0NxtoIcTEjkRiD9v9zfvmWITk309_jCYlKOkuMAEJUnDWhk-SydK1fJxA085PKUTy45g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Night Raid</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Ghosteen</td><td class="halign-left valign-top">5:08</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kE5ZilESJ9ZIjecVBlwbH7GaoRzYM1pXro2UZzWDFFGbiAos8hx8Iwqutc7tWORJ60uM25mL7xMAjtUu=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Carnage</td><td class="halign-left valign-top">Nick Cave</td><td class="halign-left valign-top"><span class="caps">CARNAGE</span></td><td class="halign-left valign-top">4:48</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kE5ZilESJ9ZIjecVBlwbH7GaoRzYM1pXro2UZzWDFFGbiAos8hx8Iwqutc7tWORJ60uM25mL7xMAjtUu=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">White Elephant</td><td class="halign-left valign-top">Nick Cave</td><td class="halign-left valign-top"><span class="caps">CARNAGE</span></td><td class="halign-left valign-top">6:09</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/tdG06VqUIgn7UB0NxtoIcTEjkRiD9v9zfvmWITk309_jCYlKOkuMAEJUnDWhk-SydK1fJxA085PKUTy45g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Ghosteen</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Ghosteen</td><td class="halign-left valign-top">12:11</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kE5ZilESJ9ZIjecVBlwbH7GaoRzYM1pXro2UZzWDFFGbiAos8hx8Iwqutc7tWORJ60uM25mL7xMAjtUu=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Lavender Fields</td><td class="halign-left valign-top">Nick Cave</td><td class="halign-left valign-top"><span class="caps">CARNAGE</span></td><td class="halign-left valign-top">4:34</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/tdG06VqUIgn7UB0NxtoIcTEjkRiD9v9zfvmWITk309_jCYlKOkuMAEJUnDWhk-SydK1fJxA085PKUTy45g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Waiting for You</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Ghosteen</td><td class="halign-left valign-top">3:55</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/pB_v5pIkHyKP-Is3uaJ61WVhTjfwc3xjBtordWeUjUqb1YaDq_ScOaQQDx0ofIJivTukOcHpi13YGngs=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">I Need You</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Skeleton Tree</td><td class="halign-left valign-top">5:59</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/cQ3CnZtnDzicg3BcE0PalKlAYq27lfvI03Fqq4Y4eozvcfruynRFM8z_hkkZbEdZPs3dRB_TNkxcQNmp=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Cosmic Dancer</td><td class="halign-left valign-top">Nick Cave</td><td class="halign-left valign-top">Cosmic Dancer</td><td class="halign-left valign-top">5:55</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/RmcC4JOjvH270v0UFftc2kyRiK-ZbxT8IhkRMi75AECQOFB2tmORyNizuXZoUFeg68ghJglx8FaGtFGq=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">God Is In The House (2011 Remastered Version)</td><td class="halign-left valign-top">Nick Cave and The Bad Seeds</td><td class="halign-left valign-top">No More Shall We Part (2011 Remastered Version)</td><td class="halign-left valign-top">5:45</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kE5ZilESJ9ZIjecVBlwbH7GaoRzYM1pXro2UZzWDFFGbiAos8hx8Iwqutc7tWORJ60uM25mL7xMAjtUu=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Hand of God</td><td class="halign-left valign-top">Nick Cave</td><td class="halign-left valign-top"><span class="caps">CARNAGE</span></td><td class="halign-left valign-top">5:17</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kE5ZilESJ9ZIjecVBlwbH7GaoRzYM1pXro2UZzWDFFGbiAos8hx8Iwqutc7tWORJ60uM25mL7xMAjtUu=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Shattered Ground</td><td class="halign-left valign-top">Nick Cave</td><td class="halign-left valign-top"><span class="caps">CARNAGE</span></td><td class="halign-left valign-top">5:35</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/Hb0nm6G0G0Zk3ZyzP85-nSKFGYoaJc8JsHHmNMJmYn3kE-RNjQxVgYsqAcZTU3sBRKZ2wXBoKbbgpppJ=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Galleon Ship (Live at Alexandra Palace, 2020)</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Galleon Ship (Live at Alexandra Palace, 2020)</td><td class="halign-left valign-top">3:32</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/tdG06VqUIgn7UB0NxtoIcTEjkRiD9v9zfvmWITk309_jCYlKOkuMAEJUnDWhk-SydK1fJxA085PKUTy45g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Leviathan</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Ghosteen</td><td class="halign-left valign-top">4:48</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/kE5ZilESJ9ZIjecVBlwbH7GaoRzYM1pXro2UZzWDFFGbiAos8hx8Iwqutc7tWORJ60uM25mL7xMAjtUu=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Balcony Man</td><td class="halign-left valign-top">Nick Cave</td><td class="halign-left valign-top"><span class="caps">CARNAGE</span></td><td class="halign-left valign-top">4:31</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/tdG06VqUIgn7UB0NxtoIcTEjkRiD9v9zfvmWITk309_jCYlKOkuMAEJUnDWhk-SydK1fJxA085PKUTy45g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Hollywood</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Ghosteen</td><td class="halign-left valign-top">14:13</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/fkv4nC0u07T3Nba6RfLW4xVDwr_cPpYmg3loq8RToZsD9Y5qaFcd7YBL05DpB3R-wCWy_LkQ8PpAXHsT=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Henry Lee [2011 Remastered Version] (feat. <span class="caps">PJ</span> Harvey)</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Murder Ballads (2011 Remastered Version)</td><td class="halign-left valign-top">3:59</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/pB_v5pIkHyKP-Is3uaJ61WVhTjfwc3xjBtordWeUjUqb1YaDq_ScOaQQDx0ofIJivTukOcHpi13YGngs=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Girl In Amber</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Skeleton Tree</td><td class="halign-left valign-top">4:52</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/hkHACooMS4NG0lz45fAqtavgyx_OZeTqT4iaqHEXtsWpq5T38h6lX2QeljGRjE0m2Ko5UmPloE_6RTpvBA=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Into My Arms (2011 Remastered Version)</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">The Boatman’s Call (2011 Remastered Version)</td><td class="halign-left valign-top">4:17</td></tr><tr><td class="halign-left valign-top"><img src="https://lh3.googleusercontent.com/tdG06VqUIgn7UB0NxtoIcTEjkRiD9v9zfvmWITk309_jCYlKOkuMAEJUnDWhk-SydK1fJxA085PKUTy45g=w60-h60-l90-rj" alt="thumbnail" width="40" height="40" class="bare">Ghosteen Speaks</td><td class="halign-left valign-top">Nick Cave <span class="amp">&</span> The Bad Seeds</td><td class="halign-left valign-top">Ghosteen</td><td class="halign-left valign-top">4:03</td></tr></tbody></table></div>
<hr>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/review-nick-cave-and-warren-ellis-gig-vancouver-2022/PXL_20220317_040117318.MP.avif" alt="Photo from my seat, showing a few rows of seating, then the stage. On stage is Nick Cave at a piano, with blue backlighting, spotlit with eight tight spotlights, alternating yellow & purple. Behind him, you can just see the gospel-style backing singers.">
<figcaption>Figure 1. The view from my seat - what a privilege to be there, what an amazing evening.</figcaption></figure></section>The Python Black formatter outputs to stderr, not stdout2022-03-06T11:50:23-08:002022-03-06T11:50:23-08:00Duncan Locktag:duncanlock.net,2022-03-06:/blog/2022/03/06/the-python-black-formatter-outputs-to-stderr-not-stdout/<p>Which is fine - it’s reporting errors, after all.
So, if you want to capture the output to a file in Linux/Bash, you need to do it like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>black <span class="nt">--check</span> <span class="nb">.</span> 2> black-report.txt</code></pre></div>
<p>or like this to <em>also</em> output to the console, so you can see it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>black <span class="nt">--check</span> <span class="nb">.</span> 2> <span class="o">>(</span><span class="nb">tee</span> <span class="nt">-a</span> black-report.txt <span class="o">></span>&2<span class="o">)</span></code></pre></div>
<p>Bash has a shortcut for this, but you lose the return code from black:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>black <span class="nt">--check</span> <span class="nb">.</span> |& <span class="nb">tee </span>black-report.txt</code></pre></div>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://stackoverflow.com/questions/692000/how-do-i-write-standard-error-to-a-file-while-using-tee-with-a-pipe">https://stackoverflow.com/questions/692000/how-do-i-write-standard-error-to-a-file-while-using-tee-with-a-pipe</a></li></ul></div></section>Windows PowerShell aliases can’t have parameters, you need to write a function.2022-03-05T09:25:57-08:002022-03-30T19:21:57-07:00Duncan Locktag:duncanlock.net,2022-03-05:/blog/2022/03/05/windows-powershell-aliases-cant-have-parameters-you-need-to-write-a-function/<p>On linux, you can create aliases for commonly used commands. These can be pretty much anything you like. Here’s an example, which creates a command called <code>ports</code>, which runs the stuff in quotes:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># List all the ports that are currently open, with the processes that are listening</span>
<span class="nb">alias </span><span class="nv">ports</span><span class="o">=</span><span class="s2">"sudo netstat -tulpn | grep '^tcp' | awk '{print </span><span class="se">\$</span><span class="s2">1,</span><span class="se">\$</span><span class="s2">4,</span><span class="se">\$</span><span class="s2">7}' | sort -n | uniq | column -t"</span></code></pre></div>
<p>On Windows, PowerShell <em>does</em> support aliases, but <em>doesn’t</em> support parameters - they’re intended to just be single PowerShell <code>Verb-Noun</code> things, I think?</p>
<p>Anyway, if you want to run a command with some parameters, you have to create a function.</p>
<p>Here’s an example of creating a simple shortcut for <code>git status</code>:</p>
<section class="doc-section level-1"><h2 id="_bash_version">Bash version:</h2><div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># git status shortcut</span>
<span class="nb">alias </span><span class="nv">gst</span><span class="o">=</span><span class="s1">'git status'</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_powershell_version">PowerShell version:</h2><div class="listing-block"><pre class="rouge highlight"><code data-lang="powershell"><span class="kr">function</span><span class="w"> </span><span class="nf">GitStatus</span><span class="w"> </span><span class="p">{</span><span class="n">git</span><span class="w"> </span><span class="nx">status</span><span class="p">}</span><span class="w">
</span><span class="n">New-Alias</span><span class="w"> </span><span class="nt">-Force</span><span class="w"> </span><span class="nt">-Name</span><span class="w"> </span><span class="nx">gst</span><span class="w"> </span><span class="nt">-Value</span><span class="w"> </span><span class="nx">GitStatus</span></code></pre></div>
<section class="doc-section level-2"><h3 id="_where_are_powershell_aliases_stored">Where are …</h3></section></section><p>On linux, you can create aliases for commonly used commands. These can be pretty much anything you like. Here’s an example, which creates a command called <code>ports</code>, which runs the stuff in quotes:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># List all the ports that are currently open, with the processes that are listening</span>
<span class="nb">alias </span><span class="nv">ports</span><span class="o">=</span><span class="s2">"sudo netstat -tulpn | grep '^tcp' | awk '{print </span><span class="se">\$</span><span class="s2">1,</span><span class="se">\$</span><span class="s2">4,</span><span class="se">\$</span><span class="s2">7}' | sort -n | uniq | column -t"</span></code></pre></div>
<p>On Windows, PowerShell <em>does</em> support aliases, but <em>doesn’t</em> support parameters - they’re intended to just be single PowerShell <code>Verb-Noun</code> things, I think?</p>
<p>Anyway, if you want to run a command with some parameters, you have to create a function.</p>
<p>Here’s an example of creating a simple shortcut for <code>git status</code>:</p>
<section class="doc-section level-1"><h2 id="_bash_version">Bash version:</h2><div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># git status shortcut</span>
<span class="nb">alias </span><span class="nv">gst</span><span class="o">=</span><span class="s1">'git status'</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_powershell_version">PowerShell version:</h2><div class="listing-block"><pre class="rouge highlight"><code data-lang="powershell"><span class="kr">function</span><span class="w"> </span><span class="nf">GitStatus</span><span class="w"> </span><span class="p">{</span><span class="n">git</span><span class="w"> </span><span class="nx">status</span><span class="p">}</span><span class="w">
</span><span class="n">New-Alias</span><span class="w"> </span><span class="nt">-Force</span><span class="w"> </span><span class="nt">-Name</span><span class="w"> </span><span class="nx">gst</span><span class="w"> </span><span class="nt">-Value</span><span class="w"> </span><span class="nx">GitStatus</span></code></pre></div>
<section class="doc-section level-2"><h3 id="_where_are_powershell_aliases_stored">Where are PowerShell aliases stored?</h3><p>PowerShell aliases are stored in various places, but this is probably the file you want to edit: <code>$Home\[My ]Documents\PowerShell\Microsoft.PowerShell_profile.ps1</code>. This is also stored in the <code>$PROFILE</code> variable:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="powershell"><span class="n">notepad</span><span class="w"> </span><span class="bp">$PROFILE</span></code></pre></div></section></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_aliases?view=powershell-7.2">https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_aliases?view=powershell-7.2</a></li><li><a class="bare" href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_profiles?view=powershell-7.2#the-profile-files">https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_profiles?view=powershell-7.2#the-profile-files</a></li></ul></div></section>Templating JSON data into a Variable, in Bash2022-02-24T22:40:50-08:002022-02-24T22:40:50-08:00Duncan Locktag:duncanlock.net,2022-02-24:/blog/2022/02/24/templating-json-data-into-variable-bash/<p>Building a <span class="caps">JSON</span> string in Bash can be pretty fiddly and annoying, with quoting and formatting and variable substitution, etc…​</p>
<p>If you only have a <em>little</em> <span class="caps">JSON</span>, using a <code>printf</code> template works well, like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c">#!/usr/bin/env bash</span>
<span class="c"># Your printf template string</span>
<span class="nv">template</span><span class="o">=</span><span class="s1">'{ "state": "%s", "key": "%s", "description": "%s", "url": "%s" }'</span>
<span class="c"># Variable to store your rendered template JSON string</span>
<span class="nv">data</span><span class="o">=</span><span class="s2">""</span>
<span class="c"># Some variables to substitute into the template</span>
<span class="nv">BITBUCKET_STATE</span><span class="o">=</span><span class="s2">"FAILED"</span>
<span class="nv">BITBUCKET_KEY</span><span class="o">=</span><span class="s2">"ci/gitlab-ci/failure"</span>
<span class="nv">BITBUCKET_DESCRIPTION</span><span class="o">=</span><span class="s2">"The build failed."</span>
<span class="nv">CI_PROJECT_URL</span><span class="o">=</span><span class="s2">"https://gitlab.example.com/project_name/repo_name"</span>
<span class="nv">CI_JOB_ID</span><span class="o">=</span><span class="s2">"123"</span>
<span class="c"># Render the template, substituting the variable values and save the result into $data</span>
<span class="nb">printf</span> <span class="nt">-v</span> data <span class="s2">"</span><span class="nv">$template</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$BITBUCKET_STATE</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$BITBUCKET_KEY</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$BITBUCKET_DESCRIPTION</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$CI_PROJECT_URL</span><span class="s2">/-/jobs/</span><span class="nv">$CI_JOB_ID</span><span class="s2">"</span>
<span class="c"># Print it out</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$data</span><span class="s2">"</span></code></pre></div>
<p>This outputs the following:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="go">{ "state": "FAILED", "key": "ci/gitlab-ci …</span></code></pre></div><p>Building a <span class="caps">JSON</span> string in Bash can be pretty fiddly and annoying, with quoting and formatting and variable substitution, etc…​</p>
<p>If you only have a <em>little</em> <span class="caps">JSON</span>, using a <code>printf</code> template works well, like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c">#!/usr/bin/env bash</span>
<span class="c"># Your printf template string</span>
<span class="nv">template</span><span class="o">=</span><span class="s1">'{ "state": "%s", "key": "%s", "description": "%s", "url": "%s" }'</span>
<span class="c"># Variable to store your rendered template JSON string</span>
<span class="nv">data</span><span class="o">=</span><span class="s2">""</span>
<span class="c"># Some variables to substitute into the template</span>
<span class="nv">BITBUCKET_STATE</span><span class="o">=</span><span class="s2">"FAILED"</span>
<span class="nv">BITBUCKET_KEY</span><span class="o">=</span><span class="s2">"ci/gitlab-ci/failure"</span>
<span class="nv">BITBUCKET_DESCRIPTION</span><span class="o">=</span><span class="s2">"The build failed."</span>
<span class="nv">CI_PROJECT_URL</span><span class="o">=</span><span class="s2">"https://gitlab.example.com/project_name/repo_name"</span>
<span class="nv">CI_JOB_ID</span><span class="o">=</span><span class="s2">"123"</span>
<span class="c"># Render the template, substituting the variable values and save the result into $data</span>
<span class="nb">printf</span> <span class="nt">-v</span> data <span class="s2">"</span><span class="nv">$template</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$BITBUCKET_STATE</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$BITBUCKET_KEY</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$BITBUCKET_DESCRIPTION</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$CI_PROJECT_URL</span><span class="s2">/-/jobs/</span><span class="nv">$CI_JOB_ID</span><span class="s2">"</span>
<span class="c"># Print it out</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$data</span><span class="s2">"</span></code></pre></div>
<p>This outputs the following:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="go">{ "state": "FAILED", "key": "ci/gitlab-ci/failure", "description": "The build failed.", "url": "https://gitlab.example.com/project_name/repo_name/-/jobs/123" }</span></code></pre></div>
<p>If you want it multi-line, you can just put newlines in the template - but then your <code>$data</code> variable will have newlines in, so be careful passing it on the command line, etc…​</p>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://stackoverflow.com/questions/48470049/build-a-json-string-with-bash-variables">https://stackoverflow.com/questions/48470049/build-a-json-string-with-bash-variables</a></li></ul></div></section>Three Editor Use Cases2021-11-13T11:14:33-08:002023-05-23T23:08:24-07:00Duncan Locktag:duncanlock.net,2021-11-13:/blog/2021/11/13/three-editor-use-cases/<p class="lead">I have three different text editors, that I use for three distinct use cases.</p>
<p>I’m a software developer, so I basically edit text files for a living, and the editing software that I use for this is fairly important to me.</p>
<section class="doc-section level-1"><h2 id="_projects">Projects</h2><p>For any substantial editing, either for work, or for personal projects, I currently use <a href="https://code.visualstudio.com/">Visual Studio Code</a> - generally known as “vscode”. It’s good enough: it’s <em>extremely</em> actively developed, so always up to date, and it has all the plug-ins you could ever want. Performance is <em>good enough</em>, once it’s started up.</p>
<p>I tend to just start my project editor once and leave it open permanently.</p>
<p>I’m still aggrieved that <a href="https://github.blog/2022-06-08-sunsetting-atom/">Microsoft killed the Atom editor</a> when they bought GitHub, despite promising that they wouldn’t. It was a better editor - better <span class="caps">UX</span>, <a href="https://tree-sitter.github.io/tree-sitter/">Tree-sitter</a>, etc …</p></section><p class="lead">I have three different text editors, that I use for three distinct use cases.</p>
<p>I’m a software developer, so I basically edit text files for a living, and the editing software that I use for this is fairly important to me.</p>
<section class="doc-section level-1"><h2 id="_projects">Projects</h2><p>For any substantial editing, either for work, or for personal projects, I currently use <a href="https://code.visualstudio.com/">Visual Studio Code</a> - generally known as “vscode”. It’s good enough: it’s <em>extremely</em> actively developed, so always up to date, and it has all the plug-ins you could ever want. Performance is <em>good enough</em>, once it’s started up.</p>
<p>I tend to just start my project editor once and leave it open permanently.</p>
<p>I’m still aggrieved that <a href="https://github.blog/2022-06-08-sunsetting-atom/">Microsoft killed the Atom editor</a> when they bought GitHub, despite promising that they wouldn’t. It was a better editor - better <span class="caps">UX</span>, <a href="https://tree-sitter.github.io/tree-sitter/">Tree-sitter</a>, etc…​</p>
<aside class="sidebar"><p>I used to use Atom for this, until Microsoft killed it - and <a href="https://www.sublimetext.com/">SublimeText</a> before that. I have a paid license for SublimeText and still try it out occasionally. The fact that it’s closed source, coupled with literally years of radio silence from the developer between updates - and the sparse/non-existent documentation - made it hard for the plugin ecosystem to get off the ground, despite the editor being great. I think they’ve improved this now, but too little too late for me.</p></aside></section>
<section class="doc-section level-1"><h2 id="_occasional_or_one_off_editing">Occasional or One-Off Editing</h2><p>If I’m just doing a quick edit of a config file or something - I generally don’t use the project editor. It’s too heavy, too slow to start and will remember all the previous open stuff - which I want in project mode, but not for one-off editing.</p>
<p>Because I have less baggage and attachment to this editor, this is where I experiment with new editors.</p>
<p>I have used <a href="https://docs.xfce.org/apps/mousepad/start">Mousepad</a>, <a href="https://cudatext.github.io/">CudaText</a>, <a href="https://lite-xl.com/">Lite <span class="caps">XL</span></a> - but currently mostly using <a href="https://lapce.dev/">Lapce</a>. Really fast startup, cross-platform, tree-sitter, good enough.</p></section>
<section class="doc-section level-1"><h2 id="_cli_editing"><span class="caps">CLI</span> Editing</h2><p>I mostly use a desktop <span class="caps">GUI</span>, but always have terminals open all over the place and often do some editing there. I’ve never learned to sing the song of vi - and I don’t feel the need, at this point. I use the excellent <a href="https://github.com/zyedidia/micro">micro editor</a> for all my terminal editing needs. I have this set in my <code>.bashrc</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="shell"><span class="nb">export </span><span class="nv">EDITOR</span><span class="o">=</span>micro
<span class="nb">export </span><span class="nv">MICRO_TRUECOLOR</span><span class="o">=</span>1</code></pre></div>
<p>Micro is <em>excellent</em>, open source and written by <a href="https://duncanlock.net/blog/2023/05/25/thanks-zachary-yedidia-zyedidia/">Zachary Yedidia</a>.</p>
<p>For me, editing in the <span class="caps">CLI</span> is generally one-off editing too. If it turns into a project, I switch to using my project editor.</p></section>Styleable Inline SVG Icons, with Caching & Fallback2021-11-09T13:08:46-08:002021-11-09T13:08:46-08:00Duncan Locktag:duncanlock.net,2021-11-09:/blog/2021/11/09/styleable-inline-svg-icon-sprite-system-with-caching-fallback/<figure class="image-block"><img alt="Tag Icon in the shape of a luggage tag" height="300px" src="https://duncanlock.net/images/icons/fa/solid/tag.svg" width="300px"/>
<figcaption>Figure 1. This pages tag list is using <a href="https://fontawesome.com/v5.15/icons/tag?style=solid">this tag icon from FontAwesome</a>. This is a 550 byte <span class="caps">SVG</span> file, 346 bytes gzipped.</figcaption></figure>
<p>If you want to use <span class="caps">SVG</span> icons on a website <em>and</em> style them with <span class="caps">CSS</span> - then the <span class="caps">SVG</span> needs to be <em>inline</em> - i.e. the <span class="caps">SVG</span> markup needs to be included with the rest of the pages <span class="caps">HTML</span> markup.</p>
<p>Unfortunately putting things inline means that they can’t be cached. In this article I’ll show one way to get around this - and get the best of both worlds: inline styleable <span class="caps">SVG</span> icons, <em>with caching!</em></p><figure class="image-block"><img alt="Tag Icon in the shape of a luggage tag" height="300px" src="https://duncanlock.net/images/icons/fa/solid/tag.svg" width="300px"/>
<figcaption>Figure 1. This pages tag list is using <a href="https://fontawesome.com/v5.15/icons/tag?style=solid">this tag icon from FontAwesome</a>. This is a 550 byte <span class="caps">SVG</span> file, 346 bytes gzipped.</figcaption></figure>
<p>If you want to use <span class="caps">SVG</span> icons on a website <em>and</em> style them with <span class="caps">CSS</span> - then the <span class="caps">SVG</span> needs to be <em>inline</em> - i.e. the <span class="caps">SVG</span> markup needs to be included with the rest of the pages <span class="caps">HTML</span> markup.</p>
<p>Unfortunately putting things inline means that they can’t be cached. In this article I’ll show one way to get around this - and get the best of both worlds: inline styleable <span class="caps">SVG</span> icons, <em>with caching!</em></p>
<p>If we take the tag list in the sidebar in this site as an example, the basic markup looks like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><section></span>
<span class="nt"><h3></span>Tags:<span class="nt"></h3></span>
<span class="nt"><ul</span> <span class="na">class=</span><span class="s">"tag-list"</span><span class="nt">></span>
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"tag"</span><span class="nt">></span>
<span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">></span> <span class="c"><!-- Tag icon goes here --></span> <span class="nt"></i></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"/tag/howto.html"</span><span class="nt">></span>howto<span class="nt"></a></span>
<span class="nt"></li></span>
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"tag"</span><span class="nt">></span>
<span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">></span> <span class="c"><!-- Tag icon goes here --></span> <span class="nt"></i></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"/tag/javascript.html"</span><span class="nt">></span>javascript<span class="nt"></a></span>
<span class="nt"></li></span>
...
<span class="nt"></ul></span>
<span class="nt"></section></span></code></pre></div>
<p>When we put the <span class="caps">SVG</span> for the icon in, it looks like this:</p>
<div class="listing-block scrollable"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><section></span>
<span class="nt"><h3</span> <span class="na">class=</span><span class="s">"tag-heading"</span><span class="nt">></span>Tags:<span class="nt"></h3></span>
<span class="nt"><ul</span> <span class="na">class=</span><span class="s">"tag-list"</span><span class="nt">></span>
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"tag"</span><span class="nt">></span>
<span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">></span>
<span class="nt"><svg</span> <span class="na">xmlns=</span><span class="s">"http://www.w3.org/2000/svg"</span> <span class="na">viewBox=</span><span class="s">"0 0 512 512"</span><span class="nt">></span><span class="c"><!-- Font Awesome Free 5.15.3 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) --></span><span class="nt"><path</span> <span class="na">d=</span><span class="s">"M0 252.118V48C0 21.49 21.49 0 48 0h204.118a48 48 0 0 1 33.941 14.059l211.882 211.882c18.745 18.745 18.745 49.137 0 67.882L293.823 497.941c-18.745 18.745-49.137 18.745-67.882 0L14.059 286.059A48 48 0 0 1 0 252.118zM112 64c-26.51 0-48 21.49-48 48s21.49 48 48 48 48-21.49 48-48-21.49-48-48-48z"</span><span class="nt">/></svg></span>
<span class="nt"></i></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"/tag/howto.html"</span><span class="nt">></span>howto<span class="nt"></a></span>
<span class="nt"></li></span>
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"tag"</span><span class="nt">></span>
<span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">></span>
<span class="nt"><svg</span> <span class="na">xmlns=</span><span class="s">"http://www.w3.org/2000/svg"</span> <span class="na">viewBox=</span><span class="s">"0 0 512 512"</span><span class="nt">></span><span class="c"><!-- Font Awesome Free 5.15.3 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) --></span><span class="nt"><path</span> <span class="na">d=</span><span class="s">"M0 252.118V48C0 21.49 21.49 0 48 0h204.118a48 48 0 0 1 33.941 14.059l211.882 211.882c18.745 18.745 18.745 49.137 0 67.882L293.823 497.941c-18.745 18.745-49.137 18.745-67.882 0L14.059 286.059A48 48 0 0 1 0 252.118zM112 64c-26.51 0-48 21.49-48 48s21.49 48 48 48 48-21.49 48-48-21.49-48-48-48z"</span><span class="nt">/></svg></span>
<span class="nt"></i></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"/tag/javascript.html"</span><span class="nt">></span>javascript<span class="nt"></a></span>
<span class="nt"></li></span>
...
<span class="nt"></ul></span>
<span class="nt"></section></span></code></pre></div>
<section class="doc-section level-1"><h2 id="_styling_inline_svg_with_css">Styling Inline <span class="caps">SVG</span> with <span class="caps">CSS</span></h2><p>As you can see, this <span class="caps">SVG</span> markup is fully inline with the rest of the <span class="caps">HTML</span>. This means that we can write <span class="caps">CSS</span> styles that will style the icon, just like we can for anything else in the <span class="caps">DOM</span>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nt">i</span><span class="nc">.icon</span> <span class="nt">svg</span> <span class="p">{</span>
<span class="c">/* Set the SVGs fill color to match the parent elements color: */</span>
<span class="py">fill</span><span class="p">:</span> <span class="n">currentColor</span><span class="p">;</span>
<span class="c">/* Set some other properties on the SVG element */</span>
<span class="nl">display</span><span class="p">:</span> <span class="n">inline-block</span><span class="p">;</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="nb">inherit</span><span class="p">;</span>
<span class="nl">height</span><span class="p">:</span> <span class="m">1em</span><span class="p">;</span>
<span class="nl">width</span><span class="p">:</span> <span class="m">1.25em</span><span class="p">;</span>
<span class="nl">overflow</span><span class="p">:</span> <span class="nb">visible</span><span class="p">;</span>
<span class="nl">vertical-align</span><span class="p">:</span> <span class="m">-0.125em</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">0</span> <span class="m">4px</span> <span class="m">0</span> <span class="m">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">/* Set different icon colours in different places, using CSS variables: */</span>
<span class="nd">:is</span><span class="o">(</span><span class="nt">header</span><span class="o">,</span> <span class="nt">footer</span><span class="o">)</span> <span class="nt">i</span><span class="nc">.icon</span> <span class="nt">svg</span> <span class="p">{</span>
<span class="py">fill</span><span class="p">:</span> <span class="n">var</span><span class="p">(</span><span class="n">--icon-color-blueprint</span><span class="p">);</span>
<span class="p">}</span>
<span class="nt">article</span> <span class="nt">i</span><span class="nc">.icon</span> <span class="nt">svg</span> <span class="p">{</span>
<span class="py">fill</span><span class="p">:</span> <span class="n">var</span><span class="p">(</span><span class="n">--icon-color-content</span><span class="p">);</span>
<span class="p">}</span>
<span class="c">/* Have icons in the header & footer change colour on hover: */</span>
<span class="nd">:is</span><span class="o">(</span><span class="nt">header</span><span class="o">,</span> <span class="nt">footer</span><span class="o">)</span> <span class="nt">a</span><span class="nd">:hover</span> <span class="nt">i</span><span class="nc">.icon</span> <span class="nt">svg</span><span class="o">,</span>
<span class="nd">:is</span><span class="o">(</span><span class="nt">header</span><span class="o">,</span> <span class="nt">footer</span><span class="o">)</span> <span class="nc">.active</span> <span class="nt">i</span><span class="nc">.icon</span> <span class="nt">svg</span> <span class="p">{</span>
<span class="py">fill</span><span class="p">:</span> <span class="n">var</span><span class="p">(</span><span class="n">--blueprint-bg</span><span class="p">);</span>
<span class="p">}</span></code></pre></div>
<aside class="sidebar"><h6 class="block-title">You can also style individual parts of the <span class="caps">SVG</span>, if you want, just like you would <span class="caps">HTML</span>:</h6><p>You can also give parts of your <span class="caps">SVG</span> classes <span class="amp">&</span> IDs, and the style these just like you would an <span class="caps">HTML</span> element with a class or <span class="caps">ID</span>. Imagine this example markup:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">></span>
<span class="nt"><svg</span> <span class="na">width=</span><span class="s">"120"</span> <span class="na">height=</span><span class="s">"120"</span> <span class="na">viewBox=</span><span class="s">"0 0 120 120"</span> <span class="na">xmlns=</span><span class="s">"http://www.w3.org/2000/svg"</span><span class="nt">></span>
<span class="nt"><rect</span> <span class="na">id=</span><span class="s">"smallRect"</span> <span class="na">x=</span><span class="s">"10"</span> <span class="na">y=</span><span class="s">"10"</span> <span class="na">width=</span><span class="s">"100"</span> <span class="na">height=</span><span class="s">"100"</span> <span class="nt">/></span>
<span class="nt"></svg></span>
<span class="nt"></i></span></code></pre></div>
<p>With this example <span class="caps">CSS</span>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nt">i</span><span class="nc">.icon</span> <span class="nt">svg</span> <span class="nt">rect</span> <span class="p">{</span> <span class="py">stroke</span><span class="p">:</span> <span class="m">#000</span><span class="p">;</span> <span class="py">fill</span><span class="p">:</span> <span class="no">red</span><span class="p">;</span> <span class="p">}</span>
<span class="nt">i</span><span class="nc">.icon</span> <span class="nt">svg</span> <span class="nf">#smallRect</span> <span class="p">{</span> <span class="py">fill</span><span class="p">:</span> <span class="no">blue</span><span class="p">;</span> <span class="p">}</span></code></pre></div>
<p>the rectangle would be blue.</p></aside></section>
<section class="doc-section level-1"><h2 id="_problems_with_inline_svg">Problems with Inline <span class="caps">SVG</span></h2><p>There are some obvious problems with doing <span class="caps">SVG</span> icons this way:</p>
<div class="dlist"><dl><dt>It’s more markup</dt><dd>Instead of keeping your <span class="caps">SVG</span> images in separate files like you would with other image files, you need to include them in your page markup.</dd><dt>It’s repetitive</dt><dd>Every time you want to use the same icon again, you have to include its markup, again. In the example list above, this is a list of tags - so it repeats the tag icon’s markup for each item in the list.</dd><dt>None of this is cached</dt><dd>Unlike external files, the browser can’t cache the actual page that’s requested, which means that the <span class="caps">SVG</span>’s aren’t cached.</dd></dl></div>
<p>The first two aren’t really a problem if you’re using some kind of templating system to generate your markup - but they both make the last one - caching, even more of a problem. The biggest drawback to inline <span class="caps">SVG</span> is that it’s not cached by the browser, it’s loaded every single time, along with the rest of the page’s markup.</p>
<p>Even if each icon is only 500 bytes, with our simple tag list which uses that icon only four times, you’re already up to 2 <span class="caps">KB</span> of extra markup. Including the header <span class="amp">&</span> footer there are currently 22 icons on this page, which even at only 500 bytes each, makes 11 <span class="caps">KB</span> of uncached extra markup on every page load. This isn’t <em>huge</em> - but it’s not ideal either. Can we do better?</p></section>
<section class="doc-section level-1"><h2 id="_using_svg_sprites_still_uncached_but_better">Using <span class="caps">SVG</span> Sprites - Still Uncached, but Better</h2><p>You can improve this by using an <span class="caps">SVG</span> “sprite sheet”:<a class="footnote-ref" href="#_footnote_1" id="_footnoteref_1" role="doc-noteref" title="View footnote 1">[1]</a>. This means putting all the <span class="caps">SVG</span> markup for all your icons into <em>one</em> <span class="caps">SVG</span> file, adding <em>that</em> inline in the top of the page somewhere, and then referencing the icons where you want to use them.</p>
<p>The sprite sheet looks like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><body></span>
<span class="c"><!-- Sprite Sheet, loaded inline somewhere at the top, but hidden: --></span>
<span class="nt"><svg</span> <span class="na">xmlns=</span><span class="s">"http://www.w3.org/2000/svg"</span> <span class="na">style=</span><span class="s">"display: none;"</span><span class="nt">></span>
<span class="nt"><symbol</span> <span class="na">id=</span><span class="s">"home"</span> <span class="na">viewBox=</span><span class="s">"0 0 512 512"</span><span class="nt">></span>
<span class="nt"><path</span> <span class="na">d=</span><span class="s">"..."</span><span class="nt">/></span>
<span class="nt"></symbol></span>
<span class="nt"><symbol</span> <span class="na">id=</span><span class="s">"tag"</span> <span class="na">viewBox=</span><span class="s">"0 0 512 512"</span><span class="nt">></span>
<span class="nt"><path</span> <span class="na">d=</span><span class="s">"..."</span><span class="nt">></span>
<span class="nt"></symbol></span>
...
<span class="nt"></svg></span></code></pre></div>
<p>When you want to use one of the icons, just reference it with an <span class="caps">SVG</span> <code>use</code> statement, and it’ll be pulled from the sprite sheet:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">></span>
<span class="nt"><svg><use</span> <span class="na">href=</span><span class="s">"#tag"</span><span class="nt">></use></svg></span>
<span class="nt"></i></span></code></pre></div>
<p>So, using the sprite sheet, we only include the markup for each icon once, then each time we want to show that icon, we include a tiny <span class="caps">SVG</span> use statement <a class="footnote-ref" href="#_footnote_2" id="_footnoteref_2" role="doc-noteref" title="View footnote 2">[2]</a>. This means that our example tag list now looks like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><ul</span> <span class="na">class=</span><span class="s">"tag-list"</span><span class="nt">></span>
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"tag"</span><span class="nt">></span>
<span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">><svg><use</span> <span class="na">href=</span><span class="s">"#tag"</span><span class="nt">></use></svg></i></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"/tag/howto.html"</span><span class="nt">></span>howto<span class="nt"></a></span>
<span class="nt"></li></span>
<span class="nt"><li</span> <span class="na">class=</span><span class="s">"tag"</span><span class="nt">></span>
<span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">><svg><use</span> <span class="na">href=</span><span class="s">"#tag"</span><span class="nt">></use></svg></i></span>
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"/tag/javascript.html"</span><span class="nt">></span>javascript<span class="nt"></a></span>
<span class="nt"></li></span>
...
<span class="nt"></ul></span></code></pre></div>
<p>This is obviously much less repetitive and less markup overall - we’re only including each icons markup once now.</p>
<p>However, the sprite sheet <em>still</em> needs to be inline if we want to be able to style the icons with <span class="caps">CSS</span> and <code><use></code> them, so it’s <em>still</em> loaded every time and <em>still</em> not cached.</p>
<p>Can we do better?</p></section>
<section class="doc-section level-1"><h2 id="_caching_inline_svg_using_svg_injection">Caching Inline <span class="caps">SVG</span> using <span class="caps">SVG</span> Injection</h2><p>If we want caching, we have to load the <span class="caps">SVG</span> sprite sheet as an external resource, like we would a normal image - but if we do that, it’s not inline anymore, so we lose the ability to style the <span class="caps">SVG</span> with <span class="caps">CSS</span>.</p>
<p>Is there a way to get the best of both worlds, somehow?</p>
<p>Yes there is, if you use some JavaScript to put your externally loaded <span class="caps">SVG</span> <em>back into the <span class="caps">DOM</span> and re-inline it</em>. Doing that looks like this:</p>
<section class="doc-section level-2"><h3 id="_1_preload_the_sprite_sheet">1. Preload the sprite sheet</h3><p>Preload <a class="footnote-ref" href="#_footnote_3" id="_footnoteref_3" role="doc-noteref" title="View footnote 3">[3]</a> the sprite sheet in your site’s <code><head></code> somewhere, ensuring that it’s loaded early and will be in the cache, ready for the next stage. This reduces jank and makes things more reliable. You can preload the sprite sheet like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><link</span> <span class="na">rel=</span><span class="s">"preload"</span> <span class="na">href=</span><span class="s">"/images/icons/icon_sheet.svg"</span> <span class="na">as=</span><span class="s">"image"</span> <span class="na">type=</span><span class="s">"image/svg+xml"</span> <span class="nt">/></span></code></pre></div></section>
<section class="doc-section level-2"><h3 id="_2_load_the_javascript">2. Load the JavaScript</h3><p>Load this JavaScript, somewhere in your site’s <code><head></code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="javascript"><span class="kd">const</span> <span class="nx">convertSVG</span> <span class="o">=</span> <span class="p">(</span><span class="nx">image</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// Get the SVG file from the cache</span>
<span class="nx">fetch</span><span class="p">(</span><span class="nx">image</span><span class="p">.</span><span class="nx">src</span><span class="p">,</span> <span class="p">{</span> <span class="na">cache</span><span class="p">:</span> <span class="dl">'</span><span class="s1">force-cache</span><span class="dl">'</span> <span class="p">})</span>
<span class="p">.</span><span class="nx">then</span><span class="p">((</span><span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="nx">res</span><span class="p">.</span><span class="nx">text</span><span class="p">())</span>
<span class="p">.</span><span class="nx">then</span><span class="p">((</span><span class="nx">data</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// Parse the SVG text and turn it into DOM nodes</span>
<span class="kd">const</span> <span class="nx">parser</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">DOMParser</span><span class="p">()</span>
<span class="kd">const</span> <span class="nx">svg</span> <span class="o">=</span> <span class="nx">parser</span>
<span class="p">.</span><span class="nx">parseFromString</span><span class="p">(</span><span class="nx">data</span><span class="p">,</span> <span class="dl">'</span><span class="s1">image/svg+xml</span><span class="dl">'</span><span class="p">)</span>
<span class="p">.</span><span class="nx">querySelector</span><span class="p">(</span><span class="dl">'</span><span class="s1">svg</span><span class="dl">'</span><span class="p">)</span>
<span class="c1">// Pass along any class or IDs from the parent <img> element</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">image</span><span class="p">.</span><span class="nx">id</span><span class="p">)</span> <span class="nx">svg</span><span class="p">.</span><span class="nx">id</span> <span class="o">=</span> <span class="nx">image</span><span class="p">.</span><span class="nx">id</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">image</span><span class="p">.</span><span class="nx">className</span><span class="p">)</span> <span class="nx">svg</span><span class="p">.</span><span class="nx">classList</span> <span class="o">=</span> <span class="nx">image</span><span class="p">.</span><span class="nx">classList</span>
<span class="c1">// Replace the parent <img> with our inline SVG</span>
<span class="nx">image</span><span class="p">.</span><span class="nx">parentNode</span><span class="p">.</span><span class="nx">replaceChild</span><span class="p">(</span><span class="nx">svg</span><span class="p">,</span> <span class="nx">image</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">callback</span><span class="p">)</span>
<span class="p">.</span><span class="k">catch</span><span class="p">((</span><span class="nx">error</span><span class="p">)</span> <span class="o">=></span> <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">error</span><span class="p">))</span>
<span class="p">}</span></code></pre></div>
<p>This converts an <span class="caps">SVG</span> image element into inline <span class="caps">SVG</span>. This allows the inline <span class="caps">SVG</span> to function normally, be styled, etc… but to <em>also be loaded async and cached by the browser as an external resource!</em>.</p>
<p>Notice this line: <code>fetch(image.src, { cache: 'force-cache' })</code> - this performs an <span class="caps">AJAX</span> load of the <span class="caps">SVG</span> file in JavaScript - but it <em>won’t be loaded twice</em> - it’ll already be in the cache and the <code>cache: 'force-cache'</code> option makes <code>fetch</code> just pull it from there.</p>
<p>Including all the comments, this is 934 bytes of JavaScript. Minified <span class="amp">&</span> gzipped, this is about 250 bytes.</p>
<aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>I didn’t invent this <span class="caps">SVG</span> injection technique, I found it here: <a class="bare" href="https://github.com/iconfu/svg-inject">https://github.com/iconfu/svg-inject</a> This is much more capable, has more features and is more compatible with older browsers than my script above. It’s also 4.5 <span class="caps">KB</span> of JavaScript - and I didn’t want any of the extra stuff, so I cribbed a very stripped down version, which is what you see here. If you want the full version, use their script instead.</p></aside></section>
<section class="doc-section level-2"><h3 id="_3_load_the_sprite_sheet_as_a_regular_image">3. Load the Sprite Sheet as a Regular Image</h3><p>Now, we can load the sprite sheet as a regular image, using an <code><img></code> element, hide it, and run the <code>convertSVG()</code> function on it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><body></span>
<span class="nt"><img</span> <span class="na">src=</span><span class="s">"/images/icons/icon_sheet.svg"</span> <span class="na">style=</span><span class="s">"display: none;"</span> <span class="na">onload=</span><span class="s">"convertSVG(this)"</span> <span class="nt">/></span>
...</code></pre></div>
<p>This is all the essential parts - we get externally loaded and cached <span class="caps">SVG</span> icon sprites, inline, with caching - the holy grail!</p>
<p>The only real downside of this technique are that it requires JavaScript. Can we have a fallback, so that this works <span class="caps">OK</span> <em>without</em> JavaScript?</p></section></section>
<section class="doc-section level-1"><h2 id="_fallback_without_javascript">Fallback Without JavaScript</h2><p>If we <em>don’t</em> have JavaScript the <code>ConvertSVG()</code> stuff isn’t going to happen. The browser will still load the sprite sheet, but the <code><use></code> stuff and the <span class="caps">CSS</span> styles won’t work, because the <span class="caps">SVG</span> will be external.</p>
<p>We can use the venerable <code><noscript></code> element <a class="footnote-ref" href="#_footnote_4" id="_footnoteref_4" role="doc-noteref" title="View footnote 4">[4]</a> to provide an alternative. The contents of the <code><noscript></code> element will only be processed by the browser if JavaScript is disabled. We can include the relevant icon as an <code><img></code> element inside the noscript element, like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><i</span> <span class="na">class=</span><span class="s">"icon"</span><span class="nt">></span>
<span class="nt"><noscript></span>
<span class="nt"><img</span> <span class="na">src=</span><span class="s">"/images/icons/fa/solid/tag.svg"</span> <span class="na">width=</span><span class="s">"18px"</span> <span class="nt">/></span>
<span class="nt"></noscript></span>
<span class="nt"><svg><use</span> <span class="na">href=</span><span class="s">"#tag"</span><span class="nt">></use></svg></span>
<span class="nt"></i></span></code></pre></div>
<p>So, if you have JavaScript enabled:</p>
<div class="ulist"><ul><li>the sprite sheet and the <code>convertSVG()</code> will work</li><li>the <code><svg><use href="#tag"></use></svg></code> will work</li><li><span class="caps">CSS</span> styles will work</li><li><code><noscript></code> element will be ignored</li></ul></div>
<p>If you have JavaScript disabled, you’ll get the opposite:</p>
<div class="ulist"><ul><li>the sprite sheet and the <code>convertSVG()</code> won’t work</li><li>the <code><svg><use href="#tag"></use></svg></code> won’t work</li><li><span class="caps">CSS</span> styles won’t work</li><li><code><noscript></code> element will be loaded <span class="amp">&</span> you’ll get the <span class="caps">SVG</span> icon loaded in the right place</li></ul></div>
<p>So, with JavaScript disabled, the icons will load in the correct place - the only thing missing will be the <span class="caps">CSS</span> styles, because the icons are now external. There’s nothing we can do about that, sadly. If you need to, you can apply some fallback styles to fix spacing issues that might arise because the intended styles no longer apply:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nt">i</span><span class="nc">.icon</span> <span class="nt">noscript</span> <span class="nt">img</span> <span class="p">{</span>
<span class="nl">padding</span><span class="p">:</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">border</span><span class="p">:</span> <span class="nb">none</span><span class="p">;</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">none</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">box-shadow</span><span class="p">:</span> <span class="nb">none</span><span class="p">;</span>
<span class="nl">margin-right</span><span class="p">:</span> <span class="m">-14px</span><span class="p">;</span>
<span class="p">}</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_show_me_the_numbers">Show Me the Numbers</h2><figure class="table-block"><figcaption>Table 1. All figures are for this page, uncompressed, and in bytes unless otherwise specified. Other external assets that this page loads, like <span class="caps">CSS</span>, have been removed for simplicity.</figcaption>
<table class="frame-all grid-all stretch"><colgroup><col style="width: 25%;"/><col style="width: 12.5%;"/><col style="width: 12.5%;"/><col style="width: 12.5%;"/><col style="width: 12.5%;"/><col style="width: 12.5%;"/><col style="width: 12.5%;"/></colgroup><thead><tr><th class="halign-left valign-top"></th><th class="halign-right valign-top"><span class="caps">HTML</span></th><th class="halign-right valign-top">JavaScript</th><th class="halign-right valign-top">Sprite Sheet</th><th class="halign-right valign-top">Total Cacheable</th><th class="halign-right valign-top">Total</th><th class="halign-right valign-top">Total Saving (%age)</th></tr></thead><tbody><tr><td class="halign-left valign-top">All Inline</td><td class="halign-right valign-top">58,793</td><td class="halign-right valign-top">-</td><td class="halign-right valign-top">-</td><td class="halign-right valign-top">-</td><td class="halign-right valign-top">58,793</td><td class="halign-right valign-top">-</td></tr><tr><td class="halign-left valign-top">Inline Sprite Sheet</td><td class="halign-right valign-top">55,940</td><td class="halign-right valign-top">-</td><td class="halign-right valign-top">-</td><td class="halign-right valign-top">-</td><td class="halign-right valign-top">55,940</td><td class="halign-right valign-top">2,853 (5%)</td></tr><tr><td class="halign-left valign-top">External Sprite Sheet, w. <span class="caps">JS</span></td><td class="halign-right valign-top">44,825</td><td class="halign-right valign-top">391</td><td class="halign-right valign-top">11,301</td><td class="halign-right valign-top">11,692</td><td class="halign-right valign-top">56,517</td><td class="halign-right valign-top">13,968 (24%)</td></tr></tbody></table></figure>
<p>We can see that that just consolidating all the repeated <span class="caps">SVG</span> icons into an inline sprite sheet saves us 2,853 bytes, or 5% of the total non-cacheable size.</p>
<p>Once you make the sprite sheet external and add the JavaScript, your overall total size actually goes up <em>very slightly</em> - by 577 bytes. However, <em>11,692 bytes of this is now cacheable</em>, where it wasn’t before - so even though you are +577 bytes on first load, you are -13,968 bytes on subsequent page loads because of caching. This is a saving of 24% over the original non-cacheable size.</p>
<hr/>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References</h3></section></section><section aria-label="Footnotes" class="footnotes" role="doc-endnotes"><hr/><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote"><span class="caps">CSS</span> Tricks Icon System with <span class="caps">SVG</span> Sprites: <a class="bare" href="https://css-tricks.com/svg-sprites-use-better-icon-fonts/">https://css-tricks.com/svg-sprites-use-better-icon-fonts/</a> <span class="amp">&</span> using <span class="caps">SVG</span> symbols: <a class="bare" href="https://css-tricks.com/svg-symbol-good-choice-icons/">https://css-tricks.com/svg-symbol-good-choice-icons/</a> <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">The <use> element takes nodes from within the <span class="caps">SVG</span> document, and duplicates them somewhere else: <a class="bare" href="https://developer.mozilla.org/en-US/docs/Web/SVG/Element/use">https://developer.mozilla.org/en-<span class="caps">US</span>/docs/Web/<span class="caps">SVG</span>/Element/use</a> <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_3" role="doc-endnote">The preload value of the <link> element’s rel attribute lets you declare fetch requests in the <span class="caps">HTML</span>’s <head>, specifying resources that your page will need very soon, which you want to start loading early in the page lifecycle, before browsers’ main rendering machinery kicks in. This ensures they are available earlier and are less likely to block the page’s render, improving performance: <a class="bare" href="https://developer.mozilla.org/en-US/docs/Web/HTML/Link_types/preload">https://developer.mozilla.org/en-<span class="caps">US</span>/docs/Web/<span class="caps">HTML</span>/Link_types/preload</a> <a class="footnote-backref" href="#_footnoteref_3" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_4" role="doc-endnote">The <noscript> <span class="caps">HTML</span> element defines a section of <span class="caps">HTML</span> to be inserted if a script type on the page is unsupported or if scripting is currently turned off in the browser: <a class="bare" href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/noscript">https://developer.mozilla.org/en-<span class="caps">US</span>/docs/Web/<span class="caps">HTML</span>/Element/noscript</a> <a class="footnote-backref" href="#_footnoteref_4" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>You Should Write to Your UK MP about Corruption Now2021-11-06T08:45:00-07:002021-11-06T08:45:00-07:00Duncan Locktag:duncanlock.net,2021-11-06:/blog/2021/11/06/you-should-write-to-your-uk-mp-about-corruption-now/<p>I’ve just emailed my <span class="caps">UK</span> <span class="caps">MP</span> to <a href="https://www.bbc.com/news/explainers-59147044">complain about government corruption <span class="amp">&</span> Owen Patterson</a>. You should do the same. As always, we need as many people as possible to speak out.</p>
<p>This is what I wrote, if anyone wants some ideas:</p>
<div class="quote-block"><blockquote><p><My Name><br>
<My Address><br>
<My Email Address></p>
<p>Dear <<span class="caps">MP</span> Name>,</p>
<p>Like many others, I am angered - but not very surprised - by the latest revelations of government corruption, revealed by the corrupt behaviour of Owen Patterson.</p>
<p>I do have to admit to being slightly surprised that the government’s reaction to this was naked collusion and a blatant attempt to water down the rules <span class="amp">&</span> standards!</p>
<p>The existing Committee on Standards needs to be strengthened, not weakened by the introduction of another Conservative-dominated committee overseeing its work.</p>
<p>The government needs to adopt the recommendations of numerous independent bodies - such as the …</p></blockquote></div><p>I’ve just emailed my <span class="caps">UK</span> <span class="caps">MP</span> to <a href="https://www.bbc.com/news/explainers-59147044">complain about government corruption <span class="amp">&</span> Owen Patterson</a>. You should do the same. As always, we need as many people as possible to speak out.</p>
<p>This is what I wrote, if anyone wants some ideas:</p>
<div class="quote-block"><blockquote><p><My Name><br>
<My Address><br>
<My Email Address></p>
<p>Dear <<span class="caps">MP</span> Name>,</p>
<p>Like many others, I am angered - but not very surprised - by the latest revelations of government corruption, revealed by the corrupt behaviour of Owen Patterson.</p>
<p>I do have to admit to being slightly surprised that the government’s reaction to this was naked collusion and a blatant attempt to water down the rules <span class="amp">&</span> standards!</p>
<p>The existing Committee on Standards needs to be strengthened, not weakened by the introduction of another Conservative-dominated committee overseeing its work.</p>
<p>The government needs to adopt the recommendations of numerous independent bodies - such as the anti-corruption watchdog the Committee on Standards in Public Life - that rules about MPs’ behaviour should be tightened.</p>
<p>As a voter in your constituency, this issue is important to me - and the governments actions so far are not good enough!</p>
<p>Yours sincerely,</p>
<p><Your Name></p></blockquote></div>
<p>This form makes it easy to email your <span class="caps">MP</span> about this issue: <a class="bare" href="https://do.opendemocracy.net/MPstandards">https://do.opendemocracy.net/MPstandards</a>. It’ll take 5 minutes - and it’s better than <em>just</em> being annoyed.</p>
<section class="doc-section level-1"><h2 id="_getting_in_touch_with_your_mp">Getting in Touch with your <span class="caps">MP</span></h2><p>Getting in touch with your <span class="caps">MP</span> generally is much easier than it used to be. You can use www.theyworkforyou.com to find your <span class="caps">UK</span> <span class="caps">MP</span> and see what they’ve been up to - as well as sending them a message.</p>
<div class="ulist"><ul><li>Does your <span class="caps">MP</span> represent you?: <a class="bare" href="https://www.theyworkforyou.com/">https://www.theyworkforyou.com/</a></li><li>Write to your politicians, national or local, for free: <a class="bare" href="https://www.writetothem.com/">https://www.writetothem.com/</a></li></ul></div></section>A ‘git up’ alias that works for any default branch2021-11-01T16:46:11-07:002021-11-01T16:46:11-07:00Duncan Locktag:duncanlock.net,2021-11-01:/blog/2021/11/01/git-up-alias-that-works-for-any-default-branch/<section id="preamble" aria-label="Preamble"><p>Traditionally, git’s default branch was called <code>master</code> - but this isn’t always true, it’s just a default - and one that’s currently undergoing change. GitHub <span class="amp">&</span> GitLab have changed their defaults to <code>main</code> and the git project itself is in the process of doing the same.</p>
<p>I have a longstanding git alias - <code>git up</code> - that I use to pull my local checkout up to date with the remote. It’s a shorthand for the following:</p>
<div class="ulist"><ul><li>Change branch to the default branch</li><li>Fetch all changes from the remote, removing any local branches that have been deleted on the remote</li><li>Pull any changes from the default remote branch</li></ul></div>
<p>This used to hardcode <code>master</code> as the default branch, but this was getting less and less reliable, so I figured out a better version that checks the repo’s default branch and uses …</p></section><section id="preamble" aria-label="Preamble"><p>Traditionally, git’s default branch was called <code>master</code> - but this isn’t always true, it’s just a default - and one that’s currently undergoing change. GitHub <span class="amp">&</span> GitLab have changed their defaults to <code>main</code> and the git project itself is in the process of doing the same.</p>
<p>I have a longstanding git alias - <code>git up</code> - that I use to pull my local checkout up to date with the remote. It’s a shorthand for the following:</p>
<div class="ulist"><ul><li>Change branch to the default branch</li><li>Fetch all changes from the remote, removing any local branches that have been deleted on the remote</li><li>Pull any changes from the default remote branch</li></ul></div>
<p>This used to hardcode <code>master</code> as the default branch, but this was getting less and less reliable, so I figured out a better version that checks the repo’s default branch and uses that:</p>
<figure class="listing-block"><figcaption>Add these lines to your ~/.gitconfig file</figcaption>
<pre class="rouge highlight"><code data-lang="bash"><span class="o">[</span><span class="nb">alias</span><span class="o">]</span>
head-branch <span class="o">=</span> <span class="o">!</span><span class="s2">"git remote show origin | grep 'HEAD branch' | cut -d' ' -f5"</span>
up <span class="o">=</span> <span class="o">!</span><span class="s2">"git switch </span><span class="si">$(</span>git head-branch<span class="si">)</span><span class="s2"> && git fetch --all --prune --progress && git pull"</span></code></pre></figure></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://stackoverflow.com/questions/28666357/git-how-to-get-default-branch">https://stackoverflow.com/questions/28666357/git-how-to-get-default-branch</a></li><li><a class="bare" href="https://about.gitlab.com/blog/2021/03/10/new-git-default-branch-name/">https://about.gitlab.com/blog/2021/03/10/new-git-default-branch-name/</a></li></ul></div></section>Speedrunning Computer Games History with a 6yr Old - Part 42021-09-19T14:04:34-07:002021-10-20T22:22:13-07:00Duncan Locktag:duncanlock.net,2021-09-19:/blog/2021/09/19/speedrunning-computer-games-history-with-a-6yr-old-part-4/<section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-4/IMG_20210919_231752-resized.webp" alt="Hand-drawn map of the Kingdom of Hyrule." width="500px" height="500px">
<figcaption>Figure 1. Our hand-drawn map of the Kingdom of Hyrule. I drew a 2x1 grid in Inkscape and printed it out, then we filled it in as we went along.</figcaption></figure>
<p>It’s been about six months since the last update. We’ve mostly been camping, playing outside and generally enjoying the summer - although we have also squeezed in some gaming, here and there.</p></section>
<section class="doc-section level-1"><h2 id="_favourite_games_so_far">Favourite Games So Far</h2><p>We spent almost all of our game time playing Legend of Zelda. It’s a towering achievement – a <em>truly</em> great game. I’m amazed that this thing is from 1986 – it doesn’t feel like it.</p>
<p>It’s also a great example of a <em>different kind</em> of co-op game. One where you play together by collaborating: discussing what to do <span class="amp">&</span> where to go next, how to tackle things, etc…​ We also took turns …</p></section><section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-4/IMG_20210919_231752-resized.webp" alt="Hand-drawn map of the Kingdom of Hyrule." width="500px" height="500px">
<figcaption>Figure 1. Our hand-drawn map of the Kingdom of Hyrule. I drew a 2x1 grid in Inkscape and printed it out, then we filled it in as we went along.</figcaption></figure>
<p>It’s been about six months since the last update. We’ve mostly been camping, playing outside and generally enjoying the summer - although we have also squeezed in some gaming, here and there.</p></section>
<section class="doc-section level-1"><h2 id="_favourite_games_so_far">Favourite Games So Far</h2><p>We spent almost all of our game time playing Legend of Zelda. It’s a towering achievement – a <em>truly</em> great game. I’m amazed that this thing is from 1986 – it doesn’t feel like it.</p>
<p>It’s also a great example of a <em>different kind</em> of co-op game. One where you play together by collaborating: discussing what to do <span class="amp">&</span> where to go next, how to tackle things, etc…​ We also took turns with the controller, with me often doing the “hard bits”.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-4/blue-darknut.png" alt="Original Blue Darknut sprite from Legend of Zelda." width="200px" height="200px">
<figcaption>Figure 2. Oh crap, Darknuts.</figcaption></figure>
<p>We got pretty stuck for a few weeks trying to get past the blue Darknuts in the 5th level dungeon. I <em>almost</em> caved and was <em>considering</em> editing the save game file, rather than having the kid lose interest – but it didn’t come to that.</p>
<p>We <em>eventually</em> completed it in the traditional, time-honoured way – sitting on the sofa in our underpants on Saturday morning, banging our heads against the level, until we eventually cracked it.</p>
<p>There was less reading in Zelda that I was hoping - but what little there was, the kid did, which was good practice. I did most of the cartography, but the kid did do some.</p>
<p>It was great fun and also, I think, the kid learnt something about being persistent and working towards longer term objectives.</p></section>
<section class="doc-section level-1"><h2 id="_updates_changes_to_the_system">Updates <span class="amp">&</span> Changes to the System</h2><section class="doc-section level-2"><h3 id="_2021_09_04">2021-09-04</h3><p>I thought it was about time we started on some adventure games – partly to help with reading practice, partly because they’re great.</p>
<section class="doc-section level-3"><h4 id="_scummvm">ScummVM</h4><div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Maniac_Mansion">Maniac Mansion</a> (1987)</li></ul></div></section></section>
<section class="doc-section level-2"><h3 id="_2021_09_18">2021-09-18</h3><p>We tentatively started having play-dates again - two so far. In preparation for this, I added some more co-op games. All of these games are 2 to 4 player co-op games, except for Gods <span class="amp">&</span> Populous, which <em>I</em> just wanted to play.</p>
<p>These are a little but of a leap in time from where we were, but I think we need to speed up anyway, so…​</p>
<section class="doc-section level-3"><h4 id="_nes"><span class="caps">NES</span></h4><div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Bomberman_II">Bomberman <span class="caps">II</span></a> (1991)</li><li><a href="https://en.wikipedia.org/wiki/Salamander_(video_game)">Life Force</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Chip_%27n_Dale_Rescue_Rangers_(video_game)">Chip ‘n Dale Rescue Rangers</a> (1990)</li></ul></div></section>
<section class="doc-section level-3"><h4 id="_amiga_1">Amiga <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a></h4><div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Gods_(video_game)">Gods</a> (1991)</li><li><a href="https://en.wikipedia.org/wiki/Golden_Axe">Golden Axe</a> (1989)</li><li><a href="https://en.wikipedia.org/wiki/The_Chaos_Engine">Chaos Engine</a> (1993)</li><li><a href="https://en.wikipedia.org/wiki/Populous_(video_game)">Populous</a> (1989)</li><li><a href="https://en.wikipedia.org/wiki/The_Fidelity_Chessmaster_2100">The Fidelity Chessmaster 2100</a> (1988)</li></ul></div></section></section></section>
<section class="doc-section level-1"><h2 id="_ill_report_back_later">I’ll report back later…​</h2><p>We’ve started on Maniac Mansion and that went <span class="caps">OK</span>, considering that it’s a new type of game that the kid hasn’t tried before - and I’m making him do all the reading 😉.</p>
<p>We’ve also played a few games of Bomberman, which is as ever, loads of fun. I’ll report back later on how we get on.</p>
<hr>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References</h3></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">The <a href="https://mega.nz/folder/gdozjZxL#uI5SheetsAd-NYKMeRjf2A/folder/5AgyxZCL">motherload of amiga games</a>, in <span class="caps">WHDL</span> format. <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>How to Contribute a Change to Nginx2021-08-14T08:23:54-07:002021-09-17T21:36:43-07:00Duncan Locktag:duncanlock.net,2021-08-14:/blog/2021/08/14/how-to-contribute-a-change-to-nginx/<section aria-label="Preamble" id="preamble"><figure class="image-block"><img alt="Two screenshots of the nginx Welcome page, side-by-side, showing the page before the change and afterwards. The one on the left is dark-on-light and the one on the right is light-on-dark." height="211px" src="https://duncanlock.net/images/posts/how-to-contribute-a-change-to-nginx/nginx-welcome-page-before-after.webp" width="547px"/>
<figcaption>Figure 1. The nginx “Welcome” page, before <span class="amp">&</span> after my intended change.</figcaption></figure>
<p>I wanted to add dark mode support to the default nginx “Welcome to nginx” page. This is about the simplest change I could choose to make - it’s a simple, backwards compatible, small additive change to one single <code>index.html</code> file. My <em>initial</em> version of this change looks like this, and is added to the files <code><style></code> element, in the <code><head></code> section:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="k">@media</span> <span class="p">(</span><span class="n">prefers-color-scheme</span><span class="p">:</span> <span class="n">dark</span><span class="p">)</span> <span class="p">{</span>
<span class="nt">body</span> <span class="p">{</span>
<span class="nl">background-color</span><span class="p">:</span> <span class="m">#363839</span><span class="p">;</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#d1cec9</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">a</span> <span class="p">{</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#c4c4ff</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></div>
<p>So, this is the process of getting that change from my brain, into the upstream nginx codebase. </p></section><section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/how-to-contribute-a-change-to-nginx/nginx-welcome-page-before-after.webp" alt="Two screenshots of the nginx Welcome page, side-by-side, showing the page before the change and afterwards. The one on the left is dark-on-light and the one on the right is light-on-dark." width="547px" height="211px">
<figcaption>Figure 1. The nginx “Welcome” page, before <span class="amp">&</span> after my intended change.</figcaption></figure>
<p>I wanted to add dark mode support to the default nginx “Welcome to nginx” page. This is about the simplest change I could choose to make - it’s a simple, backwards compatible, small additive change to one single <code>index.html</code> file. My <em>initial</em> version of this change looks like this, and is added to the files <code><style></code> element, in the <code><head></code> section:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="k">@media</span> <span class="p">(</span><span class="n">prefers-color-scheme</span><span class="p">:</span> <span class="n">dark</span><span class="p">)</span> <span class="p">{</span>
<span class="nt">body</span> <span class="p">{</span>
<span class="nl">background-color</span><span class="p">:</span> <span class="m">#363839</span><span class="p">;</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#d1cec9</span><span class="p">;</span>
<span class="p">}</span>
<span class="nt">a</span> <span class="p">{</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#c4c4ff</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></div>
<p>So, this is the process of getting that change from my brain, into the upstream nginx codebase. </p></section>
<section class="doc-section level-1"><h2 id="_checking_out_the_code_making_the_change">Checking out the Code <span class="amp">&</span> Making the Change</h2><p>The nginx project uses <a href="https://www.mercurial-scm.org/wiki/QuickStart">Mercurial</a>, rather than git for source code control. So you will need Mercurial installed for this. This page has <a href="https://www.mercurial-scm.org/wiki/Download">download links <span class="amp">&</span> installation info</a>. If you are using Debian/Ubuntu, you can just run this to install it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt <span class="nb">install </span>mercurial
<span class="go">// Once that's done, check to make sure
</span><span class="gp">$</span><span class="w"> </span>hg <span class="nt">--version</span>
<span class="go">Mercurial Distributed SCM (version 5.3.1)
</span><span class="c">...</span></code></pre></div>
<p>Once Mercurial is installed, you can check out the source and make the change like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="go">// Go to the folder where you want to check out the nginx source
</span><span class="gp">$</span><span class="w"> </span><span class="nb">cd</span> ~/dev
<span class="go">// Use Mercurial to clone the nginx repo
</span><span class="gp">$</span><span class="w"> </span>hg clone http://hg.nginx.org/nginx
<span class="go">// Go into the folder
</span><span class="gp">$</span><span class="w"> </span><span class="nb">cd </span>nginx
<span class="go">// Find the file you need to edit
</span><span class="gp">$</span><span class="w"> </span><span class="nb">grep</span> <span class="nt">-r</span> <span class="s1">'Welcome'</span> <span class="nb">.</span>
<span class="gp">./docs/html/index.html:<title></span>Welcome to nginx!</title>
<span class="gp">./docs/html/index.html:<h1></span>Welcome to nginx!</h1>
<span class="go">// Make the change
</span><span class="gp">$</span><span class="w"> </span><span class="nv">$EDITOR</span> ./docs/html/index.html</code></pre></div>
<p>To test this change, you can just open the <code>~/dev/nginx/docs/html/index.html</code> in a browser using “File → Open”, as it’s just static <span class="caps">HTML</span>.</p></section>
<section class="doc-section level-1"><h2 id="_contributing_the_change_upstream">Contributing the Change Upstream</h2><p>Nginx development is mailing list based, rather than Github/lab/forge/etc…​ based. The basic plan is to:</p>
<div class="olist arabic"><ol class="arabic"><li>Join the mailing list</li><li>Send an email to the list, containing a patch with the change.</li></ol></div>
<p>Follow <a href="https://mailman.nginx.org/mailman/listinfo/nginx-devel">the instructions here to join the mailing list</a>. You will receive a confirmation email. Once you’ve done that, you can send an email to the list with your change in.</p>
<p>To get your change in a format that’s ready to email, I did this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="go">// Commit the change and write a changelog
</span><span class="gp">$</span><span class="w"> </span>hg commit
<span class="go">// Export the change as a patch, ready to email
</span><span class="gp">$</span><span class="w"> </span>hg <span class="nb">export </span>tip</code></pre></div>
<p>This is the output of that, which I copied <span class="amp">&</span> pasted into an email – and sent it to <a href="mailto:nginx-devel@nginx.org">nginx-devel@nginx.org</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="diff"># HG changeset patch
# User Duncan Lock <duncan.lock@gmail.com>
# Date 1628952253 25200
# Sat Aug 14 07:44:13 2021 -0700
# Node ID 81294b370e774c792210904f710abc0a494c5c05
# Parent dda421871bc213dd2eb3da0015d6228839323583
<span class="p">Add support for dark color scheme in default index.html page
</span><span class="err">
</span><span class="p">Add a little CSS to index.html to support dark color schemes.
This will display the index page in dark colors if the user has
requested a dark color scheme in their system UI or browser, and
display the same as the previous version if not.
</span><span class="err">
</span><span class="p">See: https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme
</span><span class="err">
</span><span class="gh">diff -r dda421871bc2 -r 81294b370e77 docs/html/index.html
</span><span class="gd">--- a/docs/html/index.html Tue Aug 10 23:43:17 2021 +0300
</span><span class="gi">+++ b/docs/html/index.html Sat Aug 14 07:44:13 2021 -0700
</span><span class="p">@@ -8,6 +8,15 @@</span>
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
<span class="gi">+ @media (prefers-color-scheme: dark) {
+ body {
+ background-color: #363839;
+ color: #d1cec9;
+ }
+ a {
+ color: #c4c4ff;
+ }
+ }
</span> </style>
</head>
<body></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_back_forth_on_the_mailing_list">Back <span class="amp">&</span> Forth on the Mailing List</h2><p>This is <a href="https://mailman.nginx.org/pipermail/nginx-devel/2021-August/014217.html">my thread on the mailing lists web archive</a>, showing the back <span class="amp">&</span> forth discussion which ensued after my initial email.</p>
<p>First Maxim Dounin, one of the nginx core developers, politely said no thanks:</p>
<div class="quote-block"><blockquote><p>Thank you for the patch. I don’t think this is something we want
to customize in the example pages such as index.html, especially
given that we don’t set other colors.</p><footer>— <cite>Maxim Dounin, https://mailman.nginx.org/pipermail/nginx-devel/2021-August/014218.html</cite></footer></blockquote></div>
<p>My heart sank…​ but after a while I came back and read it again…​ “especially given that we don’t set other colors.”. I guess I shouldn’t have hardcoded those dark colors in my change – that’s fair enough. I then started looking to see if there was a way to tell the browser to just use its own built-in stylesheet for the dark-mode colors, just like it does for the regular, light-mode version.</p>
<p>A little while later, Steffan Weber replied with the answer:</p>
<div class="quote-block"><blockquote><p>You could add the following line which makes modern browsers use colors
from their built-in dark color scheme:</p>
<p><meta name=”color-scheme” content=”light dark”></p>
<p><a class="bare" href="https://web.dev/color-scheme/">https://web.dev/color-scheme/</a></p><footer>— <cite>Steffan Weber, https://mailman.nginx.org/pipermail/nginx-devel/2021-August/014220.html</cite></footer></blockquote></div>
<p>So, <a href="https://mailman.nginx.org/pipermail/nginx-devel/2021-August/014221.html">I said I’d submit a new updated patch in the morning</a>.</p></section>
<section class="doc-section level-1"><h2 id="_second_try">Second Try</h2><p>So, next morning I did some research <span class="amp">&</span> testing, then reset my repository to have another go:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>hg strip tip</code></pre></div>
<p>I then <a href="https://mailman.nginx.org/pipermail/nginx-devel/2021-August/014222.html">sent in this patch</a>, with Steffan’s suggested change, which did exactly what we wanted:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="diff"># HG changeset patch
# User Duncan Lock <duncan.lock@gmail.com>
# Date 1629049097 25200
# Sun Aug 15 10:38:17 2021 -0700
# Node ID 945d9836012ed84dea05577027a30a38e38a59f3
# Parent dda421871bc213dd2eb3da0015d6228839323583
<span class="p">Add support for dark color scheme in default index & 50x pages
</span><span class="err">
</span><span class="p">Add a meta tag to index.html & 50x.html to support dark color schemes.
This will display the index page in dark colors if the user has
requested a dark color scheme in their system UI or browser, and
display the same as the previous version if not.
</span><span class="err">
</span><span class="p">This uses the browsers built-in styles and doesn't hard code any colors or styles.
</span><span class="err">
</span><span class="gh">diff -r dda421871bc2 -r 945d9836012e docs/html/50x.html
</span><span class="gd">--- a/docs/html/50x.html Tue Aug 10 23:43:17 2021 +0300
</span><span class="gi">+++ b/docs/html/50x.html Sun Aug 15 10:38:17 2021 -0700
</span><span class="p">@@ -2,6 +2,7 @@</span>
<html>
<head>
<title>Error</title>
<span class="gi">+<meta name="color-scheme" content="light dark">
</span> <style>
body {
width: 35em;
<span class="gh">diff -r dda421871bc2 -r 945d9836012e docs/html/index.html
</span><span class="gd">--- a/docs/html/index.html Tue Aug 10 23:43:17 2021 +0300
</span><span class="gi">+++ b/docs/html/index.html Sun Aug 15 10:38:17 2021 -0700
</span><span class="p">@@ -2,6 +2,7 @@</span>
<html>
<head>
<title>Welcome to nginx!</title>
<span class="gi">+<meta name="color-scheme" content="light dark">
</span> <style>
body {
width: 35em;</code></pre></div>
<p>Then <a href="https://mailman.nginx.org/pipermail/nginx-devel/2021-August/014223.html">Maxim replied</a> saying, essentially, “better, but I’d rather use <span class="caps">CSS</span> instead of adding a meta tag” - i.e. remove the meta tag and add this <span class="caps">CSS</span>: <code>html { color-scheme: light dark; }</code>.</p>
<p>Maxim also included <em>his</em> patch which implemented this and asked me to test it.</p>
<p>I <a href="https://mailman.nginx.org/pipermail/nginx-devel/2021-August/014224.html">tested his patch, which worked great and then made a suggestion about removing the hardcoded fonts too</a>, which was declined, sadly.</p></section>
<section class="doc-section level-1"><h2 id="_done_released">Done <span class="amp">&</span> Released</h2><p>Maxim <a href="https://mailman.nginx.org/pipermail/nginx-devel/2021-August/014226.html">then committed his final version of the patch</a>…​ and that was it!</p>
<p>So, in the end, I didn’t <em>actually</em> get any of <em>my</em> code into nginx, but the process was fairly painless, everyone was polite and helpful – and the nginx welcome <span class="amp">&</span> error pages now respect your dark mode preferences. I did at least get my name in the changelog:</p>
<div class="quote-block"><blockquote><p>Dark mode support in welcome and 50x error pages.</p>
<p>Prodded by Duncan Lock.</p><footer>— <cite><a href="https://trac.nginx.org/nginx/timeline?from=08%2F16%2F21&daysback=5&authors=&changeset=on&repo-nginx=on&repo-nginx-tests=on&repo-nginx_org=on&milestone=on&ticket=on&ticket_details=on&wiki=on&update=Update#">Nginx Changelog</a></cite></footer></blockquote></div>
<p>This is what the welcome page code looks like now:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><table class="linenotable"><tbody><tr><td class="linenos gl"><pre class="lineno"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
</pre></td><td class="code"><pre><span class="cp"><!DOCTYPE html></span>
<span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><title></span>Welcome to nginx!<span class="nt"></title></span>
<span class="nt"><style></span>
<span class="hll"><span class="nt">html</span> <span class="p">{</span> <span class="py">color-scheme</span><span class="p">:</span> <span class="n">light</span> <span class="n">dark</span><span class="p">;</span> <span class="p">}</span>
</span><span class="nt">body</span> <span class="p">{</span> <span class="nl">width</span><span class="p">:</span> <span class="m">35em</span><span class="p">;</span> <span class="nl">margin</span><span class="p">:</span> <span class="m">0</span> <span class="nb">auto</span><span class="p">;</span>
<span class="nl">font-family</span><span class="p">:</span> <span class="n">Tahoma</span><span class="p">,</span> <span class="n">Verdana</span><span class="p">,</span> <span class="n">Arial</span><span class="p">,</span> <span class="nb">sans-serif</span><span class="p">;</span> <span class="p">}</span>
<span class="nt"></style></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><h1></span>Welcome to nginx!<span class="nt"></h1></span>
<span class="nt"><p></span>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.<span class="nt"></p></span>
<span class="nt"><p></span>For online documentation and support please refer to
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"http://nginx.org/"</span><span class="nt">></span>nginx.org<span class="nt"></a></span>.<span class="nt"><br/></span>
Commercial support is available at
<span class="nt"><a</span> <span class="na">href=</span><span class="s">"http://nginx.com/"</span><span class="nt">></span>nginx.com<span class="nt"></a></span>.<span class="nt"></p></span>
<span class="nt"><p><em></span>Thank you for using nginx.<span class="nt"></em></p></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</pre></td></tr></tbody></table></code></pre></div>
<p>This got released on Aug 31<sup>st</sup> 2021, as part of nginx 1.21.2 🎉</p></section>Cssmin is Unmaintained & has a Bug with Complex :is Selectors2021-08-05T22:19:08-07:002021-08-05T22:19:08-07:00Duncan Locktag:duncanlock.net,2021-08-05:/blog/2021/08/05/cssmin-is-unmaintained-and-has-a-bug-with-complex-is-selectors/<section id="preamble" aria-label="Preamble"><p>I was using <code>cssmin</code> to compress the <span class="caps">CSS</span> on this site during publishing. This has been working fine since April 2013, when I created this new iteration of the site using <a href="https://blog.getpelican.com/">Pelican</a>. Unbeknownst to me, cssmin has been unmaintained since October 2013. This didn’t cause any problems until recently, when I revamped the <span class="caps">CSS</span> and started using the new-ish <a href="https://duncanlock.net/blog/2021/07/19/the-css-is-selector-is-pretty-neat/">:is selector</a>.</p>
<p>So a selector like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nc">.content</span> <span class="nd">:is</span><span class="o">(</span><span class="nt">figure</span><span class="o">,</span> <span class="nt">img</span><span class="o">)</span> <span class="nd">:is</span><span class="o">(</span><span class="nc">.align-left</span><span class="o">,</span> <span class="nc">.align-right</span><span class="o">)</span></code></pre></div>
<p>would get the spaces removed and get output like this, breaking it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nc">.content</span><span class="nd">:is</span><span class="o">(</span><span class="nt">figure</span><span class="o">,</span> <span class="nt">img</span><span class="o">)</span><span class="nd">:is</span><span class="o">(</span><span class="nc">.align-left</span><span class="o">,</span> <span class="nc">.align-right</span><span class="o">)</span></code></pre></div>
<p>My solution was to change to <code>rcssmin</code>, literally a one character change to the <a href="https://pypi.org/project/pelican-webassets/">webassets</a> call:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="cp">{%</span> <span class="nv">assets</span> <span class="nv">filters</span><span class="o">=</span><span class="s2">"rcssmin"</span><span class="p">,</span> <span class="nv">output</span><span class="o">=</span><span class="s2">"css/style.min.css"</span><span class="p">,</span> <span class="err">...</span></code></pre></div>
<p>and then just update the python modules in the venv:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>python <span class="nt">-m</span> pip uninstall cssmin
<span class="gp">$</span><span class="w"> </span>python <span class="nt">-m</span> pip <span class="nb">install </span>rcssmin</code></pre></div></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://github.com/zacharyvoase/cssmin">https://github …</a></li></ul></div></section><section id="preamble" aria-label="Preamble"><p>I was using <code>cssmin</code> to compress the <span class="caps">CSS</span> on this site during publishing. This has been working fine since April 2013, when I created this new iteration of the site using <a href="https://blog.getpelican.com/">Pelican</a>. Unbeknownst to me, cssmin has been unmaintained since October 2013. This didn’t cause any problems until recently, when I revamped the <span class="caps">CSS</span> and started using the new-ish <a href="https://duncanlock.net/blog/2021/07/19/the-css-is-selector-is-pretty-neat/">:is selector</a>.</p>
<p>So a selector like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nc">.content</span> <span class="nd">:is</span><span class="o">(</span><span class="nt">figure</span><span class="o">,</span> <span class="nt">img</span><span class="o">)</span> <span class="nd">:is</span><span class="o">(</span><span class="nc">.align-left</span><span class="o">,</span> <span class="nc">.align-right</span><span class="o">)</span></code></pre></div>
<p>would get the spaces removed and get output like this, breaking it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nc">.content</span><span class="nd">:is</span><span class="o">(</span><span class="nt">figure</span><span class="o">,</span> <span class="nt">img</span><span class="o">)</span><span class="nd">:is</span><span class="o">(</span><span class="nc">.align-left</span><span class="o">,</span> <span class="nc">.align-right</span><span class="o">)</span></code></pre></div>
<p>My solution was to change to <code>rcssmin</code>, literally a one character change to the <a href="https://pypi.org/project/pelican-webassets/">webassets</a> call:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="cp">{%</span> <span class="nv">assets</span> <span class="nv">filters</span><span class="o">=</span><span class="s2">"rcssmin"</span><span class="p">,</span> <span class="nv">output</span><span class="o">=</span><span class="s2">"css/style.min.css"</span><span class="p">,</span> <span class="err">...</span></code></pre></div>
<p>and then just update the python modules in the venv:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>python <span class="nt">-m</span> pip uninstall cssmin
<span class="gp">$</span><span class="w"> </span>python <span class="nt">-m</span> pip <span class="nb">install </span>rcssmin</code></pre></div></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://github.com/zacharyvoase/cssmin">https://github.com/zacharyvoase/cssmin</a></li></ul></div></section>Custom Per Page CSS With Pelican2021-07-24T07:27:47-07:002021-10-02T01:38:46-07:00Duncan Locktag:duncanlock.net,2021-07-24:/blog/2021/07/24/custom-per-page-css-with-pelican/<section aria-label="Preamble" id="preamble"><p>Sometimes I want to have a just <em>a tiny bit</em> of <span class="caps">CSS</span> that’s unique, just for one page or post.
I don’t want a <em>whole</em> stylesheet, or to have to add this to my site-wide theme, just for one post – I want a simple way to add it in the post itself.</p>
<p>This is how I did it:</p></section><section id="preamble" aria-label="Preamble"><p>Sometimes I want to have a just <em>a tiny bit</em> of <span class="caps">CSS</span> that’s unique, just for one page or post.
I don’t want a <em>whole</em> stylesheet, or to have to add this to my site-wide theme, just for one post – I want a simple way to add it in the post itself.</p>
<p>This is how I did it:</p></section>
<section class="doc-section level-1"><h2 id="_in_your_theme_templates">In Your Theme Templates</h2><p>Add an <code>extrahead</code> block into your templates, inside the <span class="caps">HTML</span> <code><head></code> section, so that you can add extra stuff into the head in some templates:</p>
<section class="doc-section level-2"><h3 id="_in_base_html_j2">In base.html.j2</h3><div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="nt"><head></span>
...
<span class="cp">{%</span> <span class="k">block</span> <span class="nv">extrahead</span> <span class="cp">%}</span> <span class="cp">{%</span> <span class="k">endblock</span> <span class="cp">%}</span>
<span class="nt"></head></span></code></pre></div>
<aside class="sidebar"><p>I already had the <code>extrahead</code> block and use it for other things, like this in my archive template:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="cp">{%</span> <span class="k">extends</span> <span class="s2">"base.html.j2"</span> <span class="cp">%}</span>
...
<span class="cp">{%</span> <span class="k">block</span> <span class="nv">extrahead</span> <span class="cp">%}</span>
<span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">media=</span><span class="s">"all"</span> <span class="na">href=</span><span class="s">"</span><span class="cp">{{</span> <span class="nv">SITEURL</span> <span class="cp">}}</span><span class="s">/theme/css/archive.css"</span> <span class="nt">/></span>
<span class="cp">{%</span> <span class="k">endblock</span> <span class="cp">%}</span></code></pre></div></aside>
<p>Then, in the page/post/article templates, add a <code><style></code> into the <code>extrahead</code> if the page metadata contains an <code>extra_css</code> entry:</p></section>
<section class="doc-section level-2"><h3 id="_in_article_html_j2">In article.html.j2</h3><div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="cp">{%</span> <span class="k">extends</span> <span class="s2">"base.html.j2"</span> <span class="cp">%}</span>
...
<span class="cp">{%</span> <span class="k">block</span> <span class="nv">extrahead</span> <span class="cp">%}</span>
...
<span class="cp">{%</span> <span class="k">if</span> <span class="nv">article.extra_css</span> <span class="cp">%}</span>
<span class="nt"><style></span>
<span class="cp">{{</span> <span class="nv">article.extra_css</span> <span class="cp">}}</span>
<span class="nt"></style></span>
<span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span>
<span class="cp">{%</span> <span class="k">endblock</span> <span class="cp">%}</span></code></pre></div></section>
<section class="doc-section level-2"><h3 id="_in_page_html_j2">In page.html.j2</h3><div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="cp">{%</span> <span class="k">extends</span> <span class="s2">"base.html.j2"</span> <span class="cp">%}</span>
...
<span class="cp">{%</span> <span class="k">block</span> <span class="nv">extrahead</span> <span class="cp">%}</span>
...
<span class="cp">{%</span> <span class="k">if</span> <span class="nv">page.extra_css</span> <span class="cp">%}</span>
<span class="nt"><style></span>
<span class="cp">{{</span> <span class="nv">page.extra_css</span> <span class="cp">}}</span>
<span class="nt"></style></span>
<span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span>
<span class="cp">{%</span> <span class="k">endblock</span> <span class="cp">%}</span></code></pre></div></section></section>
<section class="doc-section level-1"><h2 id="_in_your_postpage">In Your Post/Page</h2><p>Add something like this to the post/page metadata:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="asciidoc">:extra_css: .wavy-underline { text-decoration: red wavy underline; }</code></pre></div>
<p>and then in your content itself, apply the <span class="caps">CSS</span>, something like this for AsciiDoc<a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="asciidoc">This will have a [.wavy-underline]#red wavy underline#, but this won't.</code></pre></div>
<p>something like this for Markdown, where you need to use inline <span class="caps">HTML</span>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="markdown">This will have a <span class="nt"><span</span> <span class="na">class=</span><span class="s">"wavy-underline"</span><span class="nt">></span>red wavy underline<span class="nt"></span></span>, but this won't.</code></pre></div>
<p>something like this for reStructuredText:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">:extra_css: .wavy-underline { text-decoration: red wavy underline; }
.. role: wavy-underline
This will have a :wavy-underline:`red wavy underline`, but this won't.</code></pre></div>
<p>and it looks like this:</p>
<p>This will have a <span class="wavy-underline">red wavy underline</span>, but this won’t.</p></section>
<section class="doc-section level-1"><h2 id="_references_footnotes">References <span class="amp">&</span> Footnotes</h2><div class="ulist"><ul><li>If you want to add whole custom stylesheets to pages or posts, you can use the <a href="https://notabug.org/jorgesumle/pelican-css">pelican-css plugin</a>.</li><li><a href="https://rest-sphinx-memo.readthedocs.io/en/latest/Sphinx.html#css-class">Rest <span class="amp">&</span> Sphinx Memo: Defining a css class for some part</a></li></ul></div></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">Note that this didn’t initially work with the pelican asciidoc-reader plugin - but will now that this <span class="caps">PR</span> is merged: <a class="bare" href="https://github.com/getpelican/pelican-plugins/pull/1344">https://github.com/getpelican/pelican-plugins/pull/1344</a> <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>The CSS :is selector is pretty neat2021-07-19T12:15:25-07:002021-07-19T12:15:25-07:00Duncan Locktag:duncanlock.net,2021-07-19:/blog/2021/07/19/the-css-is-selector-is-pretty-neat/<section id="preamble" aria-label="Preamble"><p>I found out about the <span class="caps">CSS</span> <code>:is</code> selector today, via this <a href="https://blog.jim-nielsen.com/2021/things-i-learned-reading-webkits-ua-stylesheet/">“Things I Learned Reading Webkit’s <span class="caps">UA</span> Stylesheet”</a> article.</p>
<p>It takes a selector list as its argument, and selects any element that can be selected by one of the selectors in that list. This is useful for writing large selectors in a more compact form:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Instead of this: */</span>
<span class="nc">.content</span> <span class="nt">p</span> <span class="o">></span> <span class="nt">code</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">td</span> <span class="o">></span> <span class="nt">code</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">li</span> <span class="o">></span> <span class="nt">code</span> <span class="p">{</span>
<span class="c">/* ... */</span>
<span class="p">}</span>
<span class="c">/* You can do this: */</span>
<span class="nc">.content</span> <span class="nd">:is</span><span class="o">(</span><span class="nt">p</span><span class="o">,</span> <span class="nt">td</span><span class="o">,</span> <span class="nt">li</span><span class="o">)</span> <span class="o">></span> <span class="nt">code</span> <span class="p">{</span>
<span class="c">/* ... */</span>
<span class="p">}</span></code></pre></div>
<p>you can also put it in the middle of selectors, like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Instead of this: */</span>
<span class="nc">.content</span> <span class="nt">h1</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">h2</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">h3</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">h4</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">h5</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">hr</span> <span class="p">{</span>
<span class="c">/* ... */</span>
<span class="p">}</span>
<span class="c">/* You can do this: */</span>
<span class="nc">.content</span> <span class="nd">:is</span><span class="o">(</span><span class="nt">h1</span><span class="o">,</span> <span class="nt">h2</span><span class="o">,</span> <span class="nt">h3</span><span class="o">,</span> <span class="nt">h4</span><span class="o">,</span> <span class="nt">h5</span><span class="o">,</span> <span class="nt">hr</span><span class="o">)</span> <span class="p">{</span>
<span class="c">/* ... */</span>
<span class="p">}</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://developer.mozilla.org/en-US/docs/Web/CSS/:is">https://developer.mozilla.org/en-<span class="caps">US</span>/docs/Web/<span class="caps">CSS</span>/:is</a></li><li><a class="bare" href="https://www.caniuse.com/?search=is">https://www.caniuse.com/?search=is</a></li><li><a class="bare" href="https://blog.jim-nielsen.com/2021/things-i-learned-reading-webkits-ua-stylesheet/">https://blog.jim-nielsen.com/2021/things-i-learned-reading-webkits-ua-stylesheet …</a></li></ul></div></section><section id="preamble" aria-label="Preamble"><p>I found out about the <span class="caps">CSS</span> <code>:is</code> selector today, via this <a href="https://blog.jim-nielsen.com/2021/things-i-learned-reading-webkits-ua-stylesheet/">“Things I Learned Reading Webkit’s <span class="caps">UA</span> Stylesheet”</a> article.</p>
<p>It takes a selector list as its argument, and selects any element that can be selected by one of the selectors in that list. This is useful for writing large selectors in a more compact form:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Instead of this: */</span>
<span class="nc">.content</span> <span class="nt">p</span> <span class="o">></span> <span class="nt">code</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">td</span> <span class="o">></span> <span class="nt">code</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">li</span> <span class="o">></span> <span class="nt">code</span> <span class="p">{</span>
<span class="c">/* ... */</span>
<span class="p">}</span>
<span class="c">/* You can do this: */</span>
<span class="nc">.content</span> <span class="nd">:is</span><span class="o">(</span><span class="nt">p</span><span class="o">,</span> <span class="nt">td</span><span class="o">,</span> <span class="nt">li</span><span class="o">)</span> <span class="o">></span> <span class="nt">code</span> <span class="p">{</span>
<span class="c">/* ... */</span>
<span class="p">}</span></code></pre></div>
<p>you can also put it in the middle of selectors, like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Instead of this: */</span>
<span class="nc">.content</span> <span class="nt">h1</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">h2</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">h3</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">h4</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">h5</span><span class="o">,</span>
<span class="nc">.content</span> <span class="nt">hr</span> <span class="p">{</span>
<span class="c">/* ... */</span>
<span class="p">}</span>
<span class="c">/* You can do this: */</span>
<span class="nc">.content</span> <span class="nd">:is</span><span class="o">(</span><span class="nt">h1</span><span class="o">,</span> <span class="nt">h2</span><span class="o">,</span> <span class="nt">h3</span><span class="o">,</span> <span class="nt">h4</span><span class="o">,</span> <span class="nt">h5</span><span class="o">,</span> <span class="nt">hr</span><span class="o">)</span> <span class="p">{</span>
<span class="c">/* ... */</span>
<span class="p">}</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://developer.mozilla.org/en-US/docs/Web/CSS/:is">https://developer.mozilla.org/en-<span class="caps">US</span>/docs/Web/<span class="caps">CSS</span>/:is</a></li><li><a class="bare" href="https://www.caniuse.com/?search=is">https://www.caniuse.com/?search=is</a></li><li><a class="bare" href="https://blog.jim-nielsen.com/2021/things-i-learned-reading-webkits-ua-stylesheet/">https://blog.jim-nielsen.com/2021/things-i-learned-reading-webkits-ua-stylesheet/</a></li></ul></div></section>Don’t Put Eggs Under Your Tomatoes, if you have Raccoons2021-07-13T18:06:13-07:002021-07-13T18:06:13-07:00Duncan Locktag:duncanlock.net,2021-07-13:/blog/2021/07/13/dont-put-eggs-under-your-tomatoes-if-you-have-raccoons/<p>They will dig up your garden to get them.</p>
<p>I’ve seen various recommendations for breaking an egg into the bottom of the hole when you plant out tomatoes. Apparently, this works and increases yield:</p>
<div class="video-block"><iframe src="//www.youtube.com/embed/ZQ8Iq3qQ-h8?rel=0" frameborder="0" allowfullscreen></iframe></div>
<p>It’s just a form of fertilizer, so this isn’t too surprising. I’ve tried this in previous years and it seems to work. Sadly when I did it again this year, the raccoons decided they wanted the eggs and dug up the garden to get them.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/dont-put-eggs-under-your-tomatoes-if-you-have-raccoons/IMG_20210629_094011-smaller.webp" alt="Picture of the garden bed the morning after. Shows some plants dug up, with soild scattered around, while others are untouched.">
<figcaption>Figure 1. Part of the aftermath of the raccoon raid. They were actually <em>fairly</em> targeted, digging up all the tomato plants that had eggs underneath, with limited collateral damage.</figcaption></figure>
<p>So, if you have raccoons, don’t bury eggs – or anything else edible <span class="amp">&</span> smelly – in the garden.</p>Running python webservers on port 80 without root2021-07-11T00:24:37-07:002021-10-01T12:57:28-07:00Duncan Locktag:duncanlock.net,2021-07-11:/blog/2021/07/11/running-python-webservers-on-port-80-without-root/<section id="preamble" aria-label="Preamble"><p>If you want to have a python webserver running locally and listening on port 80 – or any other port < 1024, you can do it like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>setcap <span class="s1">'cap_net_bind_service=+ep'</span> /usr/bin/python3.8</code></pre></div>
<p>This gives the <code>/usr/bin/python3.8</code> binary the ability to bind to privileged ports without being root. Note that this only works for real files, not symlinks, so this will probably not work, as <code>python</code> is generally a symlink:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>setcap <span class="s1">'cap_net_bind_service=+ep'</span> <span class="si">$(</span>which python<span class="si">)</span><span class="sb">`</span></code></pre></div>
<p>I wanted this so that the <code>invoke livereload</code> thing for this blog, could take over from Nginx for local development, giving me incremental live rebuilding <span class="amp">&</span> reloading while editing. This happens to use tornado underneath, which is a Python webserver.</p></section>
<section class="doc-section level-1"><h2 id="_removing_capabilities">Removing Capabilities</h2><p>If you want to undo this, you do this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">#</span><span class="w"> </span>to remove just that …</code></pre></div></section><section id="preamble" aria-label="Preamble"><p>If you want to have a python webserver running locally and listening on port 80 – or any other port < 1024, you can do it like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>setcap <span class="s1">'cap_net_bind_service=+ep'</span> /usr/bin/python3.8</code></pre></div>
<p>This gives the <code>/usr/bin/python3.8</code> binary the ability to bind to privileged ports without being root. Note that this only works for real files, not symlinks, so this will probably not work, as <code>python</code> is generally a symlink:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>setcap <span class="s1">'cap_net_bind_service=+ep'</span> <span class="si">$(</span>which python<span class="si">)</span><span class="sb">`</span></code></pre></div>
<p>I wanted this so that the <code>invoke livereload</code> thing for this blog, could take over from Nginx for local development, giving me incremental live rebuilding <span class="amp">&</span> reloading while editing. This happens to use tornado underneath, which is a Python webserver.</p></section>
<section class="doc-section level-1"><h2 id="_removing_capabilities">Removing Capabilities</h2><p>If you want to undo this, you do this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">#</span><span class="w"> </span>to remove just that one capability, use <span class="nt">-ep</span>
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>setcap <span class="s1">'cap_net_bind_service=-ep'</span> /usr/bin/python3.8
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>to see that capabilities on a file
<span class="gp">$</span><span class="w"> </span>getcap /usr/bin/python3.8
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>to remove all of them
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>setcap <span class="nt">-r</span> /usr/bin/python3.8</code></pre></div></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a href="https://stackoverflow.com/questions/413807/is-there-a-way-for-non-root-processes-to-bind-to-privileged-ports-on-linux">Is there a way for non-root processes to bind to “privileged” ports on Linux?</a></li><li><a href="https://gist.github.com/firstdoit/6389682">Allow Node to bind to port 80 without sudo</a></li></ul></div></section>Ninja is a surprisingly nice build system for tiny projects2021-06-22T13:39:25-07:002021-06-22T13:39:25-07:00Duncan Locktag:duncanlock.net,2021-06-22:/blog/2021/06/22/ninja-is-a-surprisingly-nice-build-system-for-tiny-projects/<section id="preamble" aria-label="Preamble"><p><a href="https://ninja-build.org/">Ninja</a> is a surprisingly nice build system for tiny projects. That’s more or less the opposite of what it’s designed for, but it works really well for tiny things – things that are small enough that you can just create the little build file by hand. You just need to <a href="https://github.com/ninja-build/ninja/releases">install it (you can just put the binary on the $<span class="caps">PATH</span> somewhere)</a>, then create a <code>ninja.build</code> file in your folder. Here’s an example that published a couple of AsciiDoc files and re-built the index if any of the pages changed:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="slim"><span class="nt">opts</span><span class="w"> </span><span class="p">=</span> <span class="o">-</span><span class="n">a</span> <span class="n">source</span><span class="o">-</span><span class="n">highlighter</span><span class="o">=</span><span class="n">rouge</span> <span class="o">-</span><span class="n">a</span> <span class="n">rouge</span><span class="o">-</span><span class="n">style</span><span class="o">=</span><span class="n">monokai</span> <span class="o">-</span><span class="n">r</span> <span class="n">asciidoctor</span><span class="o">-</span><span class="n">html5s</span> <span class="o">-</span><span class="n">b</span> <span class="n">html5s</span>
<span class="nt">rule</span><span class="w"> </span>adoc
<span class="nt">command</span><span class="w"> </span><span class="p">=</span> <span class="n">asciidoctor</span> <span class="vg">$opts</span> <span class="vg">$in</span> <span class="o">-</span><span class="n">o</span> <span class="vg">$out</span>
<span class="nt">build</span><span class="w"> </span>out/page1<span class="nc">.html</span>:<span class="w"> </span>adoc<span class="w"> </span>src/page1<span class="nc">.adoc</span>
<span class="nt">build</span><span class="w"> </span>out/page2<span class="nc">.html</span>:<span class="w"> </span>adoc<span class="w"> </span>src/page2<span class="nc">.adoc</span>
<span class="nt">build</span><span class="w"> </span>out/index<span class="nc">.html</span>:<span class="w"> </span>adoc<span class="w"> </span>src/index<span class="nc">.adoc</span><span class="w"> </span>|<span class="w"> </span>src …</code></pre></div></section><section id="preamble" aria-label="Preamble"><p><a href="https://ninja-build.org/">Ninja</a> is a surprisingly nice build system for tiny projects. That’s more or less the opposite of what it’s designed for, but it works really well for tiny things – things that are small enough that you can just create the little build file by hand. You just need to <a href="https://github.com/ninja-build/ninja/releases">install it (you can just put the binary on the $<span class="caps">PATH</span> somewhere)</a>, then create a <code>ninja.build</code> file in your folder. Here’s an example that published a couple of AsciiDoc files and re-built the index if any of the pages changed:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="slim"><span class="nt">opts</span><span class="w"> </span><span class="p">=</span> <span class="o">-</span><span class="n">a</span> <span class="n">source</span><span class="o">-</span><span class="n">highlighter</span><span class="o">=</span><span class="n">rouge</span> <span class="o">-</span><span class="n">a</span> <span class="n">rouge</span><span class="o">-</span><span class="n">style</span><span class="o">=</span><span class="n">monokai</span> <span class="o">-</span><span class="n">r</span> <span class="n">asciidoctor</span><span class="o">-</span><span class="n">html5s</span> <span class="o">-</span><span class="n">b</span> <span class="n">html5s</span>
<span class="nt">rule</span><span class="w"> </span>adoc
<span class="nt">command</span><span class="w"> </span><span class="p">=</span> <span class="n">asciidoctor</span> <span class="vg">$opts</span> <span class="vg">$in</span> <span class="o">-</span><span class="n">o</span> <span class="vg">$out</span>
<span class="nt">build</span><span class="w"> </span>out/page1<span class="nc">.html</span>:<span class="w"> </span>adoc<span class="w"> </span>src/page1<span class="nc">.adoc</span>
<span class="nt">build</span><span class="w"> </span>out/page2<span class="nc">.html</span>:<span class="w"> </span>adoc<span class="w"> </span>src/page2<span class="nc">.adoc</span>
<span class="nt">build</span><span class="w"> </span>out/index<span class="nc">.html</span>:<span class="w"> </span>adoc<span class="w"> </span>src/index<span class="nc">.adoc</span><span class="w"> </span>|<span class="w"> </span>src/page1<span class="nc">.adoc</span><span class="w"> </span>src/page2<span class="nc">.adoc</span></code></pre></div>
<p>It’s super fast, incremental <span class="amp">&</span> idempotent – just re-run it and it’ll just redo anything that needs doing – and <em>nothing that doesn’t</em>.</p></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a href="https://ninja-build.org/manual.html#_writing_your_own_ninja_files">Writing Your Own Ninja Files</a></li><li><a href="https://jvns.ca/blog/2020/10/26/ninja–a-simple-way-to-do-builds/">ninja: a simple way to do builds</a></li></ul></div></section>Better content-type guessing in AWS CLI2021-06-21T14:59:26-07:002021-11-07T07:26:27-08:00Duncan Locktag:duncanlock.net,2021-06-21:/blog/2021/06/21/better-content-type-guessing-in-aws-cli/<p>The content-type guessing done by <span class="caps">AWS</span> <span class="caps">CLI</span> is based on the mimetype definitions available on your system. You can improve the mimetype guessing by updating these definitions.</p>
<p>It uses the python <code>mimetypes</code> lib to do this and <a href="https://github.com/python/cpython/blob/cedc9b74202d8c1ae39bca261cbb45d42ed54d45/Lib/mimetypes.py#L42-L52">accumulates its list of mimetyoes from the following files</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">knownfiles</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">"/etc/mime.types"</span><span class="p">,</span>
<span class="s">"/etc/httpd/mime.types"</span><span class="p">,</span> <span class="c1"># Mac OS X
</span> <span class="s">"/etc/httpd/conf/mime.types"</span><span class="p">,</span> <span class="c1"># Apache
</span> <span class="s">"/etc/apache/mime.types"</span><span class="p">,</span> <span class="c1"># Apache 1
</span> <span class="s">"/etc/apache2/mime.types"</span><span class="p">,</span> <span class="c1"># Apache 2
</span> <span class="s">"/usr/local/etc/httpd/conf/mime.types"</span><span class="p">,</span>
<span class="s">"/usr/local/lib/netscape/mime.types"</span><span class="p">,</span>
<span class="s">"/usr/local/etc/httpd/conf/mime.types"</span><span class="p">,</span> <span class="c1"># Apache 1.2
</span> <span class="s">"/usr/local/etc/mime.types"</span><span class="p">,</span> <span class="c1"># Apache 1.3
</span> <span class="p">]</span></code></pre></div>
<p>I would suggest leaving <code>/etc/mime.types</code> (and any of the other files in this list that already exist on your system) alone. I would create one of the files that you don’t …</p><p>The content-type guessing done by <span class="caps">AWS</span> <span class="caps">CLI</span> is based on the mimetype definitions available on your system. You can improve the mimetype guessing by updating these definitions.</p>
<p>It uses the python <code>mimetypes</code> lib to do this and <a href="https://github.com/python/cpython/blob/cedc9b74202d8c1ae39bca261cbb45d42ed54d45/Lib/mimetypes.py#L42-L52">accumulates its list of mimetyoes from the following files</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">knownfiles</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">"/etc/mime.types"</span><span class="p">,</span>
<span class="s">"/etc/httpd/mime.types"</span><span class="p">,</span> <span class="c1"># Mac OS X
</span> <span class="s">"/etc/httpd/conf/mime.types"</span><span class="p">,</span> <span class="c1"># Apache
</span> <span class="s">"/etc/apache/mime.types"</span><span class="p">,</span> <span class="c1"># Apache 1
</span> <span class="s">"/etc/apache2/mime.types"</span><span class="p">,</span> <span class="c1"># Apache 2
</span> <span class="s">"/usr/local/etc/httpd/conf/mime.types"</span><span class="p">,</span>
<span class="s">"/usr/local/lib/netscape/mime.types"</span><span class="p">,</span>
<span class="s">"/usr/local/etc/httpd/conf/mime.types"</span><span class="p">,</span> <span class="c1"># Apache 1.2
</span> <span class="s">"/usr/local/etc/mime.types"</span><span class="p">,</span> <span class="c1"># Apache 1.3
</span> <span class="p">]</span></code></pre></div>
<p>I would suggest leaving <code>/etc/mime.types</code> (and any of the other files in this list that already exist on your system) alone. I would create one of the files that you don’t have – something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo mkdir</span> <span class="nt">-p</span> /etc/apache2
<span class="gp">$</span><span class="w"> </span>curl <span class="nt">-s</span> <span class="s2">"http://svn.apache.org/viewvc/httpd/httpd/trunk/docs/conf/mime.types?view=co"</span> | <span class="nb">sudo tee</span> /etc/apache2/mime.types</code></pre></div>
<p>This downloads the latest mimetypes definitions from the Apache project and saves it as <code>/etc/apache2/mime.types</code>. I don’t have the Apache web server installed on this system, so that file wasn’t there before. It <a href="https://svn.apache.org/viewvc/httpd/httpd/trunk/docs/conf/mime.types">looks like</a> this file get updated about once a year.</p>Fixing content types in s3 using the AWS CLI2021-06-21T14:59:02-07:002021-11-07T07:18:34-08:00Duncan Locktag:duncanlock.net,2021-06-21:/blog/2021/06/21/fixing-content-types-in-s3-using-the-aws-cli/<section id="preamble" aria-label="Preamble"><p>I had an issue where the <span class="caps">AWS</span> <span class="caps">CLI</span> wasn’t guessing the content-type of <span class="caps">SVG</span> files correctly on sync and was setting them to <code>application/octet-stream</code> - the default “I don’t know” mimetype. <a href="https://duncanlock.net/blog/2021/06/21/better-content-type-guessing-in-aws-cli/">There’s a proper fix to make the mimetype guessing work here</a>. This is a quick fix for stuff that’s already been uploaded to s3:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>aws s3 <span class="nb">cp</span> <span class="nt">--exclude</span> <span class="s2">"*"</span> <span class="nt">--include</span> <span class="s2">"*.svg"</span> <span class="nt">--content-type</span><span class="o">=</span><span class="s2">"image/svg+xml"</span> <span class="nt">--metadata-directive</span><span class="o">=</span><span class="s2">"REPLACE"</span> <span class="nt">--recursive</span> <span class="nt">--acl</span> public-read ./output/ s3://<bucketname></code></pre></div>
<p>This <em>probably</em> wipes out some of the other metadata on the files, but sets <code>content-type</code> <span class="amp">&</span> <code>acl</code> correctly for uploading <span class="caps">SVG</span> files to a website hosted on s3.</p>
<aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>I found out later that this was affecting more files <span class="amp">&</span> filetypes than I thought, so the simplest fix was to delete everything in the bucket and re-upload it, after <a href="https://duncanlock.net/blog/2021/06/21/better-content-type-guessing-in-aws-cli/">fixing the <span class="caps">AWS</span> CLIs content type …</a></p></aside></section><section id="preamble" aria-label="Preamble"><p>I had an issue where the <span class="caps">AWS</span> <span class="caps">CLI</span> wasn’t guessing the content-type of <span class="caps">SVG</span> files correctly on sync and was setting them to <code>application/octet-stream</code> - the default “I don’t know” mimetype. <a href="https://duncanlock.net/blog/2021/06/21/better-content-type-guessing-in-aws-cli/">There’s a proper fix to make the mimetype guessing work here</a>. This is a quick fix for stuff that’s already been uploaded to s3:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>aws s3 <span class="nb">cp</span> <span class="nt">--exclude</span> <span class="s2">"*"</span> <span class="nt">--include</span> <span class="s2">"*.svg"</span> <span class="nt">--content-type</span><span class="o">=</span><span class="s2">"image/svg+xml"</span> <span class="nt">--metadata-directive</span><span class="o">=</span><span class="s2">"REPLACE"</span> <span class="nt">--recursive</span> <span class="nt">--acl</span> public-read ./output/ s3://<bucketname></code></pre></div>
<p>This <em>probably</em> wipes out some of the other metadata on the files, but sets <code>content-type</code> <span class="amp">&</span> <code>acl</code> correctly for uploading <span class="caps">SVG</span> files to a website hosted on s3.</p>
<aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>I found out later that this was affecting more files <span class="amp">&</span> filetypes than I thought, so the simplest fix was to delete everything in the bucket and re-upload it, after <a href="https://duncanlock.net/blog/2021/06/21/better-content-type-guessing-in-aws-cli/">fixing the <span class="caps">AWS</span> CLIs content type guessing</a>.</p></aside></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a href="https://stackoverflow.com/questions/50856831/copy-to-s3-with-aws-cli-with-proper-content-type">Copy to S3 with <span class="caps">AWS</span> <span class="caps">CLI</span> with proper content type
</a></li><li><a href="https://serverfault.com/questions/725562/recursively-changing-the-content-type-for-files-of-a-given-extension-on-amazon-s">Recursively changing the content-type for files of a given extension on Amazon S3</a></li></ul></div></section>Lessons from Two Years of Fermenting2021-06-16T20:43:05+07:002021-06-16T20:43:05+07:00Duncan Locktag:duncanlock.net,2021-06-16:/blog/2021/06/16/lessons-from-two-years-of-fermenting/<section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/lessons-from-two-years-of-fermenting/IMG_20190330_104815-smaller.webp" alt="Three mason jars, full of fermenting things. Two Bánh Mì(ish) and one eggs." width="400">
<figcaption>Figure 1. Three mason jars, full of fermenting vegetables. Two Bánh Mì(ish) and one eggs.</figcaption></figure>
<p>I’ve been fermenting vegetables of various kinds (and occasionally other things) at home for a little over two years. It’s a fun, interesting <span class="amp">&</span> nutritious hobby. On the whole it’s pretty easy and low stakes – the worst thing that happens is a few cucumbers go mouldy.</p>
<p>Because it doesn’t involve any heat – but does include pouring, measuring, stirring, spiralizing, grating (and some chopping) – it’s ideal to do with little kids.</p>
<p>According to my notes (which I didn’t start initially), I’ve made ~56 batches of ferments, ranging in size from a single 1l mason jar, to three 2l ones.</p>
<p>These are my lessons so far:</p></section>
<section class="doc-section level-1"><h2 id="_the_difference_between_science_screwing_around_is_writing_it_down">The difference between science <span class="amp">&</span> screwing around, is writing it down</h2><p>As they say, the …</p></section><section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/lessons-from-two-years-of-fermenting/IMG_20190330_104815-smaller.webp" alt="Three mason jars, full of fermenting things. Two Bánh Mì(ish) and one eggs." width="400">
<figcaption>Figure 1. Three mason jars, full of fermenting vegetables. Two Bánh Mì(ish) and one eggs.</figcaption></figure>
<p>I’ve been fermenting vegetables of various kinds (and occasionally other things) at home for a little over two years. It’s a fun, interesting <span class="amp">&</span> nutritious hobby. On the whole it’s pretty easy and low stakes – the worst thing that happens is a few cucumbers go mouldy.</p>
<p>Because it doesn’t involve any heat – but does include pouring, measuring, stirring, spiralizing, grating (and some chopping) – it’s ideal to do with little kids.</p>
<p>According to my notes (which I didn’t start initially), I’ve made ~56 batches of ferments, ranging in size from a single 1l mason jar, to three 2l ones.</p>
<p>These are my lessons so far:</p></section>
<section class="doc-section level-1"><h2 id="_the_difference_between_science_screwing_around_is_writing_it_down">The difference between science <span class="amp">&</span> screwing around, is writing it down</h2><p>As they say, the difference between science and screwing around is writing it down – so keep notes. If you want to perfect recipes <span class="amp">&</span> techniques, write down what you did and iterate. Change things which didn’t work out, ideally just changing one thing at a time, so you can see the effect.</p></section>
<section class="doc-section level-1"><h2 id="_leave_headspace_at_the_top_of_the_jar">Leave headspace at the top of the jar</h2><p>Leave a layer of clear brine, above the ferment <span class="amp">&</span> weights. For short ferments up to a month, ~1.5” or ~4 cm. For longer ferments, double it. Only the good lacto-fermenting bacteria can grow in the brine (mostly), so a layer of clear brine at the top will keep anything else from getting to your fermenting vegetables down below. Sometimes things grow on the surface (see below) – but if you have enough headspace, you can just skim it off and it won’t bother the ferments below.</p></section>
<section class="doc-section level-1"><h2 id="_keep_everything_underneath_the_brine">Keep everything underneath the brine</h2><p>Make sure nothing rises to, or floats on, the surface. No matter what you do, <em>anything</em> on the surface will go mouldy <em>eventually</em>. Simple ways to do this are to push a whole cabbage leaf down over the top of everything, or a slice of rutabaga/swede that’s about the same size as the jar – then a weight on top of that.</p></section>
<section class="doc-section level-1"><h2 id="_what_is_the_white_stuff_on_the_surface_of_the_waterwhat_is_kahm_yeast_do_i_have_it">What is the white stuff on the surface of the water/What is Kahm Yeast <span class="amp">&</span> do I have it?</h2><p>This is by far the most common question on <a href="https://www.reddit.com/r/fermentation/">/r/fermentation</a>.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/lessons-from-two-years-of-fermenting/IMG_20210122_082207.webp" alt="looking down at a white patchy layer of Kahm Yeast growing on the surface of the brine inside a wide mouth mason jar." width="300">
<figcaption>Figure 2. Kahm Yeast growing on the surface of a ferment. Yes, this is what Kahm Yeast looks like. Yes, you will get it from time to time.</figcaption></figure>
<p>Kahm Yeast looks like a thin, white, mostly flat patchy layer on the surface of the brine. It’s mostly harmless, but if you leave it too long it’ll make things taste off – skim it off with a spoon or kitchen paper.</p>
<p>Kahm yeast is white <span class="amp">&</span> fairly flat – anything colourful, furry/hairy or lumpy is mould. If you catch the mould early, skim it off and carry on. If it looks black/pink/angry or has taken over, remove it, maybe try a pickle and see what you think, but maybe compost the lot to be on the safe side.</p></section>
<section class="doc-section level-1"><h2 id="_use_wide_mouth_mason_jars_with_easy_fermenter_lids_weights">Use wide mouth mason jars, with Easy Fermenter lids <span class="amp">&</span> weights</h2><p>The ferment will produce some CO₂, causing jars/lids to burst. Either “burp” it by unscrewing the lid a little or don’t fully tighten the lid.</p>
<p>Actually forget that, just use wide-mouth mason jars <span class="amp">&</span> Easy Fermenter lids – these are <em>excellent</em>, reducing mould and increasing ease <span class="amp">&</span> reliability quite a bit: <a href="https://amzn.to/3q2qMt5">Easy Fermenter Wide Mouth Lid Kit</a></p></section>Good, simple, Bash slugify function2021-06-15T12:24:56-07:002021-06-15T12:24:56-07:00Duncan Locktag:duncanlock.net,2021-06-15:/blog/2021/06/15/good-simple-bash-slugify-function/<section id="preamble" aria-label="Preamble"><div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># Slugify</span>
<span class="c"># Transliterate everything to ASCII</span>
<span class="c"># Strip out apostrophes</span>
<span class="c"># Anything that's not a letter or number to a dash</span>
<span class="c"># Strip leading & trailing dashes</span>
<span class="c"># Everything to lowercase</span>
<span class="k">function </span>slugify<span class="o">()</span> <span class="o">{</span>
iconv <span class="nt">-t</span> ascii//TRANSLIT <span class="se">\</span>
| <span class="nb">tr</span> <span class="nt">-d</span> <span class="s2">"'"</span> <span class="se">\</span>
| <span class="nb">sed</span> <span class="nt">-E</span> <span class="s1">'s/[^a-zA-Z0-9]+/-/g'</span> <span class="se">\</span>
| <span class="nb">sed</span> <span class="nt">-E</span> <span class="s1">'s/^-+|-+$//g'</span> <span class="se">\</span>
| <span class="nb">tr</span> <span class="s2">"[:upper:]"</span> <span class="s2">"[:lower:]"</span>
<span class="o">}</span></code></pre></div>
<p>This is <a href="https://www.oilshell.org/blog/2017/01/15.html">point-free style</a> – so, once you’ve declared it, you can use it in a pipe, like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nv">title</span><span class="o">=</span><span class="s2">"This is my long Title, with some Punctuation!?"</span>
<span class="nv">title_slug</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$title</span><span class="s2">"</span> | slugify<span class="si">)</span></code></pre></div>
<p>or like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">echo</span> <span class="s2">"This is my long Title, with some Punctuation!?"</span> | slugify
<span class="go">this-is-my-long-title-with-some-punctuation</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li>Mostly modified from here: <a class="bare" href="https://gist.github.com/oneohthree/f528c7ae1e701ad990e6">https://gist.github.com/oneohthree/f528c7ae1e701ad990e6</a></li><li><a href="https://stackoverflow.com/questions/19335215/what-is-a-slug">What is a Slug?</a></li></ul></div></section>Promise.allSettled in JavaScript2021-06-15T10:34:00-07:002021-06-15T10:34:00-07:00Duncan Locktag:duncanlock.net,2021-06-15:/blog/2021/06/15/promise-allsettled-in-javascript/<section aria-label="Preamble" id="preamble"><p>Sometimes I want to resolve several promises at once, then do something when they’re all done. For example, make several <span class="caps">API</span> calls, then do something with all the results.</p></section><section id="preamble" aria-label="Preamble"><p>Sometimes I want to resolve several promises at once, then do something when they’re all done. For example, make several <span class="caps">API</span> calls, then do something with all the results.</p>
<p>I was doing something like this before:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="js"><span class="c1">// Build your promises array</span>
<span class="kd">let</span> <span class="nx">promises</span> <span class="o">=</span> <span class="p">[</span>
<span class="nx">fetch</span><span class="p">(</span><span class="dl">'</span><span class="s1">https://blah.com/static/file.json</span><span class="dl">'</span><span class="p">),</span>
<span class="nx">fetch</span><span class="p">(</span><span class="dl">'</span><span class="s1">https://blah.com/api/thing/one</span><span class="dl">'</span><span class="p">),</span>
<span class="nx">fetch</span><span class="p">(</span><span class="dl">'</span><span class="s1">https://blah.com/apr/thing/two</span><span class="dl">'</span><span class="p">),</span>
<span class="p">]</span>
<span class="nb">Promise</span><span class="p">.</span><span class="nx">all</span><span class="p">(</span><span class="nx">promises</span><span class="p">.</span><span class="nx">map</span><span class="p">((</span><span class="nx">p</span><span class="p">)</span> <span class="o">=></span> <span class="nx">p</span><span class="p">.</span><span class="k">catch</span><span class="p">((</span><span class="nx">err</span><span class="p">)</span> <span class="o">=></span> <span class="nx">err</span><span class="p">)))</span>
<span class="p">.</span><span class="nx">then</span><span class="p">(([</span><span class="nx">file</span><span class="p">,</span> <span class="nx">thing1</span><span class="p">,</span> <span class="nx">thing2</span><span class="p">])</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">file</span><span class="p">)</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">thing1</span><span class="p">)</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">thing2</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">.</span><span class="k">catch</span><span class="p">((</span><span class="nx">err</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">({</span> <span class="nx">err</span> <span class="p">})</span>
<span class="p">})</span></code></pre></div>
<p>A simpler, neater <span class="amp">&</span> easier to read version uses <code>Promise.allSettled(promises)</code> to do the same thing:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="js"><span class="c1">// Build your promises array</span>
<span class="kd">let</span> <span class="nx">promises</span> <span class="o">=</span> <span class="p">[</span>
<span class="nx">fetch</span><span class="p">(</span><span class="dl">'</span><span class="s1">https://blah.com/static/file.json</span><span class="dl">'</span><span class="p">),</span>
<span class="nx">fetch</span><span class="p">(</span><span class="dl">'</span><span class="s1">https://blah.com/api/thing/one</span><span class="dl">'</span><span class="p">),</span>
<span class="nx">fetch</span><span class="p">(</span><span class="dl">'</span><span class="s1">https://blah.com/apr/thing/two</span><span class="dl">'</span><span class="p">),</span>
<span class="p">]</span>
<span class="c1">// Resolve all the promises</span>
<span class="nb">Promise</span><span class="p">.</span><span class="nx">allSettled</span><span class="p">(</span><span class="nx">promises</span><span class="p">)</span>
<span class="p">.</span><span class="nx">then</span><span class="p">(([</span><span class="nx">file</span><span class="p">,</span> <span class="nx">thing1</span><span class="p">,</span> <span class="nx">thing2</span><span class="p">])</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// Each promise in the original array is now a response object:</span>
<span class="c1">// For each outcome object, a status string is present. If the status is fulfilled, then a value is present.</span>
<span class="c1">// If the status is rejected, then a reason is present.</span>
<span class="c1">// The value (or reason) reflects what value each promise was fulfilled (or rejected) with.</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">file</span><span class="p">)</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">thing1</span><span class="p">)</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">thing2</span><span class="p">)</span>
<span class="p">})</span>
<span class="p">.</span><span class="k">catch</span><span class="p">((</span><span class="nx">err</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">({</span> <span class="nx">err</span> <span class="p">})</span>
<span class="p">})</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a class="bare" href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled">https://developer.mozilla.org/en-<span class="caps">US</span>/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled</a></li><li><a class="bare" href="https://javascript.info/promise-api">https://javascript.info/promise-api</a></li></ul></div></section>Nullish coalescing in JavaScript (??)2021-06-09T11:31:39-07:002021-06-09T11:31:39-07:00Duncan Locktag:duncanlock.net,2021-06-09:/blog/2021/06/09/nullish-coalescing-in-javascript/<section id="preamble" aria-label="Preamble"><p>My new favourite thing in <span class="caps">JS</span> is Nullish coalescing:</p>
<div class="quote-block"><blockquote><p>The nullish coalescing operator (<code>??</code>) is a logical operator that returns its right-hand side operand when its left-hand side operand is <code>null</code> or <code>undefined</code>, and otherwise returns its left-hand side operand.</p><footer>— <cite><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Nullish_coalescing_operator">Nullish Coalescing Operator</a></cite></footer></blockquote></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="js"><span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">thing1</span> <span class="o">??</span> <span class="k">this</span><span class="p">.</span><span class="nx">thing2</span> <span class="o">??</span> <span class="dl">'</span><span class="s1">default</span><span class="dl">'</span></code></pre></div>
<p>Also works well with the <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining">Optional Chaining Operator</a>, which is also great - not more “Uncaught TypeError: Cannot read property ‘thing1’ of undefined”:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="js"><span class="k">return</span> <span class="k">this</span><span class="p">?.</span><span class="nx">things</span><span class="p">.?.</span><span class="nx">thing1</span> <span class="o">??</span> <span class="k">this</span><span class="p">?.</span><span class="nx">things</span><span class="p">?.[</span><span class="dl">'</span><span class="s1">thing2</span><span class="dl">'</span><span class="p">]</span> <span class="o">??</span> <span class="dl">'</span><span class="s1">default</span><span class="dl">'</span></code></pre></div>
<p>This is like logical or - <code>||</code> - but instead of testing for <code>truthy</code> <span class="amp">&</span> <code>falsy</code>, it tests for <code>nullish</code> - which is often what you want.</p>
<p>Nullish all the things!</p></section>
<section class="doc-section level-1"><h2 id="_references">References</h2><div class="ulist"><ul><li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Nullish_coalescing_operator">Nullish Coalescing Operator</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Glossary/Falsy">Falsy</a></li></ul></div></section>Speedrunning Computer Games History with a 6yr Old - Part 32021-04-25T10:13:02-07:002021-06-07T06:21:07-07:00Duncan Locktag:duncanlock.net,2021-04-25:/blog/2021/04/25/speedrunning-computer-games-history-with-a-6yr-old-part-3/<section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-3/Gauntlet_Atari-arcade-gameplay-screenshot-7.png" alt="Gauntlet Screenshot" width="320" height="229">
<figcaption>Figure 1. Gauntlet, by far and away the kids favourite game so far.</figcaption></figure>
<p>Time for another update on the Retro Gaming project. It’s been roughly a month since the last update and we’ve been playing a bit more often this time.</p></section>
<section class="doc-section level-1"><h2 id="_favourite_games_so_far">Favourite Games So Far</h2><p>Roughly in order of playtime:</p>
<div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Gauntlet_(1985_video_game)">Gauntlet</a> (1985)</li><li><a href="https://en.wikipedia.org/wiki/Super_Mario_Bros.">Super Mario Bros.</a> (1985)</li></ul></div>
<p>Gauntlet is <em>far and away</em> the kids favourite game so far. Almost nothing else got a look in since I put it on. <em>Even</em> Super Mario Bros is a pretty distant second place, and the other games have barely been played at all.</p>
<p>I think there are two reasons for this. Firstly, Gauntlet is a 4 player co-operative game. So far we’ve only played with two players, but even then, co-op games are just <em>really fun</em>. I think the other reason …</p></section><section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-3/Gauntlet_Atari-arcade-gameplay-screenshot-7.png" alt="Gauntlet Screenshot" width="320" height="229">
<figcaption>Figure 1. Gauntlet, by far and away the kids favourite game so far.</figcaption></figure>
<p>Time for another update on the Retro Gaming project. It’s been roughly a month since the last update and we’ve been playing a bit more often this time.</p></section>
<section class="doc-section level-1"><h2 id="_favourite_games_so_far">Favourite Games So Far</h2><p>Roughly in order of playtime:</p>
<div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Gauntlet_(1985_video_game)">Gauntlet</a> (1985)</li><li><a href="https://en.wikipedia.org/wiki/Super_Mario_Bros.">Super Mario Bros.</a> (1985)</li></ul></div>
<p>Gauntlet is <em>far and away</em> the kids favourite game so far. Almost nothing else got a look in since I put it on. <em>Even</em> Super Mario Bros is a pretty distant second place, and the other games have barely been played at all.</p>
<p>I think there are two reasons for this. Firstly, Gauntlet is a 4 player co-operative game. So far we’ve only played with two players, but even then, co-op games are just <em>really fun</em>. I think the other reason is that Gauntlet is an arcade game and Super Mario Bros isn’t. This means that Gauntlet has a <em>built-in</em> easy/cheat mode - putting in more quarters - whereas Mario doesn’t, it’s just <a href="https://en.wikipedia.org/wiki/Nintendo_hard">Nintendo Hard</a> (it’s actually not that bad by 80s standards).</p>
<p>In our case, we can put virtual quarters into our emulated arcade machine anytime we like, just by pressing “select” on the controller. This means that the kid can keep playing Gauntlet indefinitely - without “failing” or even facing much adversity.</p>
<p>This obviously removes the challenge, but it seems not to matter much to the kid - it seems like winning is winning, no matter what. To be fair, Gauntlet is also very compelling - in common with lots of games - there’s always loads happening, levels are fairly compact, so turn over quickly <span class="amp">&</span> you never get a break. Infinite health also removes a lot of the depth and strategy from the game, because you can just wade through everything without conserving health.</p></section>
<section class="doc-section level-1"><h2 id="_updates_changes_to_the_system">Updates <span class="amp">&</span> Changes to the System</h2><p>I made the following changes since the last update:</p>
<section class="doc-section level-2"><h3 id="_2021_04_23">2021-04-23:</h3><p>I added the following games:</p>
<section class="doc-section level-3"><h4 id="_arcade">Arcade</h4><div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Marble_Madness">Marble Madness</a> (1984)</li><li><a href="https://en.wikipedia.org/wiki/Asteroids_(video_game)">Asteroids</a> (1979)</li><li><a href="https://en.wikipedia.org/wiki/Blasteroids">Blasteroids</a> (1987)</li></ul></div>
<p>I went back in time a bit and put Asteroids on, because it’s a classic and we missed it - but mostly because we had a conversation about <a href="https://en.wikipedia.org/wiki/Ed_Logg">Ed Logg</a>. We generally try to talk about the people who create the things we enjoy - artists, authors, musicians, etc…​ to emphasize that these things are all made by real people. Anyway, Ed Logg co-created <a href="https://en.wikipedia.org/wiki/Asteroids_(video_game)">Asteroids</a> - but also co-created <a href="https://en.wikipedia.org/wiki/Centipede_(video_game)">Centipede</a> and designed <a href="https://en.wikipedia.org/wiki/Gauntlet_(1985_video_game)">Gauntlet</a> - two great games that we’ve both enjoyed. Blasteroids is a souped-up two player co-op version of Asteroids - and, as we know, multiplayer co-op is <em>super fun</em>.</p></section>
<section class="doc-section level-3"><h4 id="_nes"><span class="caps">NES</span></h4><div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/The_Legend_of_Zelda_(video_game)">The Legend of Zelda</a> (1986)</li></ul></div>
<p>I’m personally really looking forward to playing this, as I didn’t have a <span class="caps">NES</span> when I was a kid, so I’ve never played it before. Hoping to get the kid to do some/most of the reading - and maybe draw a map, but we’ll see how it goes.</p></section>
<section class="doc-section level-3"><h4 id="_amiga">Amiga</h4><div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Prince_of_Persia_(1989_video_game)">Prince of Persia</a> (1989)</li></ul></div>
<p>I finally got the Amiga emulator up and running! I also found a <span class="caps">PDF</span> manual for Prince of Persia which tells you the controls, which is very helpful, as there’s a lot of falling into spiked pit traps from great heights otherwise.</p></section>
<section class="doc-section level-3"><h4 id="_c64">C64</h4><div class="ulist"><ul><li><a href="https://www.c64-wiki.com/wiki/Battleships">Battleships</a></li></ul></div>
<p>Two kids were playing an electronic version of Battleships in the park, which the kid had never seen before. We had a conversation about it, so I put this on. No manual, no idea what the keys are, etc…​ Might just be easier to play it on paper.</p></section></section></section>
<section class="doc-section level-1"><h2 id="_ill_report_back_later">I’ll report back later…​</h2><p>Quite looking forward to playing some of the great Amiga games, as well as my first play of Legend of Zelda! I’ll report back later on how we got on.</p></section>Speedrunning Computer Games History with a 6yr Old - Part 22021-03-27T18:22:51-07:002021-04-04T01:17:25-07:00Duncan Locktag:duncanlock.net,2021-03-27:/blog/2021/03/27/speedrunning-computer-games-history-with-a-6yr-old-part-2/<section id="preamble" aria-label="Preamble"><p>Time for an update on the Retro Gaming project. It’s been almost four months since we started and it’s been, predictably, lots of fun.
What with work and school and everything else, we have very little time for playing computer games. That, combined with a fairly dry winter <span class="amp">&</span> spring letting us get outside more, means that even with the Covid-19 pandemic curtailing activities, we haven’t clocked very many hours of playtime.</p></section>
<section class="doc-section level-1"><h2 id="_favourite_games_so_far">Favourite Games So Far</h2><p>Roughly in order of playtime:</p>
<div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Galaga_%2788">Galaga ‘88</a> (1988)</li><li><a href="https://en.wikipedia.org/wiki/Galaxian">Galaxian</a> (1979)</li><li><a href="https://en.wikipedia.org/wiki/Pac-Man">Pac-Man</a> (1980)</li><li><a href="https://en.wikipedia.org/wiki/Defender_(1981_video_game)">Defender</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Bubble_Bobble">Bubble Bobble</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Centipede_(video_game)">Centipede</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Donkey_Kong_(video_game)">Donkey Kong</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Frogger">Frogger</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Pitfall!">Pitfall!</a> (1982)</li><li><a href="https://en.wikipedia.org/wiki/Cavern_Creatures">Cavern Creatures</a> (1983)</li></ul></div>
<p>In the end, we didn’t actually play any <span class="caps">ZX</span> Spectrum or C64 games at all, sadly. I played a lot of these when I was a kid, so I was secretly disappointed …</p></section><section id="preamble" aria-label="Preamble"><p>Time for an update on the Retro Gaming project. It’s been almost four months since we started and it’s been, predictably, lots of fun.
What with work and school and everything else, we have very little time for playing computer games. That, combined with a fairly dry winter <span class="amp">&</span> spring letting us get outside more, means that even with the Covid-19 pandemic curtailing activities, we haven’t clocked very many hours of playtime.</p></section>
<section class="doc-section level-1"><h2 id="_favourite_games_so_far">Favourite Games So Far</h2><p>Roughly in order of playtime:</p>
<div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Galaga_%2788">Galaga ‘88</a> (1988)</li><li><a href="https://en.wikipedia.org/wiki/Galaxian">Galaxian</a> (1979)</li><li><a href="https://en.wikipedia.org/wiki/Pac-Man">Pac-Man</a> (1980)</li><li><a href="https://en.wikipedia.org/wiki/Defender_(1981_video_game)">Defender</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Bubble_Bobble">Bubble Bobble</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Centipede_(video_game)">Centipede</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Donkey_Kong_(video_game)">Donkey Kong</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Frogger">Frogger</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Pitfall!">Pitfall!</a> (1982)</li><li><a href="https://en.wikipedia.org/wiki/Cavern_Creatures">Cavern Creatures</a> (1983)</li></ul></div>
<p>In the end, we didn’t actually play any <span class="caps">ZX</span> Spectrum or C64 games at all, sadly. I played a lot of these when I was a kid, so I was secretly disappointed by this, but the emulators for both of these systems are <em>far too true to life</em> - i.e. they’re a bit janky and fiddly. There are also just <em>better</em> versions of most of these games on other contemporary systems. At this point in the journey, if there’s an Arcade version, it’ll be much better (see below for more on this).</p></section>
<section class="doc-section level-1"><h2 id="_observations_so_far">Observations So Far</h2><p>The <a href="https://www.raspberrypi.org/products/raspberry-pi-400/">Raspberry Pi 400</a> is <em>really</em> great; I’m very happy with this little machine.</p>
<p>Emulator quality is <em>really</em> important.</p>
<p><a href="https://retropie.org.uk/">Retropie</a> is a really nice frontend.</p>
<p>Popular early games were ported to basically every contemporary platform, so you can just choose the best available version, or the one that works best on your emulators.</p>
<p>Pitfall is, roughly, the only good Atari 2600 game, but it’s fairly hard and the graphics are basic. The kid likes getting the little person eaten by crocodiles.</p>
<p>There’s a big visual quality gap between 1983 arcade games (comparatively good) vs 1983 home computer games (comparatively bad). The arcade greats also feel far more polished, which they probably were. Crucially, as we don’t have to put <em>actual money</em> into our emulated arcade games, this doesn’t provide any incentive to switch to the “worse” games. So, there’s a little bump in the road here.</p>
<p>There are lots of two player arcade games that are two player <em>alternating</em>. Two player <em>simultaneous</em> or co-op games are really <em>much, much more</em> fun, but rarer. Bubble Bobble is the only two player co-op we’ve played so far. I just loaded Gauntlet, which is 4 player co-op, so we should give that a go too.</p>
<p>Bubble Bobble doesn’t have anything mapped to the up arrow, but has a jump button instead. I think this would work better if jump was up, so I need to figure out how to remap <code>lr-mame</code> input.</p>
<p>We’re <em>only</em> playing the best games ever created, year by year - so saying they’re good is pretty redundant, but most of these games are also <em>much deeper</em> than they initially appear.</p>
<p>Playing everything with the <a href="https://www.nytimes.com/wirecutter/reviews/best-pc-gaming-controller/#for-retro-gamers-buffalo-classic-usb-gamepad"><span class="caps">NES</span> style controllers</a> I got off eBay is <span class="caps">OK</span>, but doesn’t work as well with the home computer titles, like the C64 <span class="amp">&</span> <span class="caps">ZX</span> Spectrum, which expect a keyboard. The Raspberry Pi 400 <em>does have</em> a keyboard, so I need to get us used to using it to play games. It’s a bit cramped for multiplayer, but then <em>so was the Spectrum</em> - and we used to do that just fine.</p>
<p>The kid isn’t a very confident reader yet, so text-heavy games haven’t worked out so well. Reading is still “work”, and games which flash text on-screen briefly are too quick to catch. Things like Zork are all text, but you proceed entirely at your own pace. These were a bit much last time we tried - and I ended up doing all the reading. They’re also hard <span class="amp">&</span> slow (without hints) so they require <em>a lot</em> of patience from a kid - or even an adult, really. The kid was intrigued by them, but I think we’ll come back to these later.</p>
<p>I need to figure out a schedule and drop games more frequently, otherwise we’re not going to get to the end before modernity puts a stop to it.</p></section>
<section class="doc-section level-1"><h2 id="_updates_changes_to_the_system">Updates <span class="amp">&</span> Changes to the System</h2><p>I made the following changes since the last update:</p>
<section class="doc-section level-2"><h3 id="_2021_02_19">2021-02-19:</h3><p>Replaced the <span class="caps">ZX</span> Spectrum version of Bubble Bobble with the arcade version, which is <em>much</em> better.</p></section>
<section class="doc-section level-2"><h3 id="_2021_03_27">2021-03-27:</h3><p>Realized that we need to speed up if we’re going to get through our tour of gaming history before Roblox <span class="amp">&</span> Minecraft sweep this all away, so I added a few more:</p>
<section class="doc-section level-3"><h4 id="_arcade">Arcade</h4><p>I missed a few great games on the initial load, so lets get them on before we get too far along:</p>
<div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Ghosts_%27n_Goblins_(video_game)">Ghosts ‘n Goblins</a> (1985)</li><li><a href="https://en.wikipedia.org/wiki/Gauntlet_(1985_video_game)">Gauntlet</a> (1985)</li><li><a href="https://en.wikipedia.org/wiki/Out_Run">Out Run</a> (1986)</li><li><a href="https://en.wikipedia.org/wiki/Tempest_(video_game)">Tempest</a> (1981)</li></ul></div></section>
<section class="doc-section level-3"><h4 id="_nes"><span class="caps">NES</span></h4><p>I didn’t have access to a <span class="caps">NES</span> when I was a kid, so I never played any of the classic Nintendo games growing up - so this should be lots of fun. I added two classic early games:</p>
<div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Super_Mario_Bros.">Super Mario Bros.</a> (1985)</li><li><a href="https://en.wikipedia.org/wiki/Tetris">Tetris</a> (1989)</li></ul></div>
<p>I also replaced the speccy Tetris with this version, because the speccy version, true to life, spent ages squealing and loading from its virtual tape file, then didn’t work. While this was somewhat amusing for me, it’s not <em>actually fun</em> for anyone. I also removed “Where in the World is Carmen Sandeigo (1985)” from the Apple <span class="caps">II</span>, as this didn’t really work well enough, because swapping disks is annoying in the emulator.</p></section></section></section>
<section class="doc-section level-1"><h2 id="_ill_report_back_later">I’ll report back later…​</h2><p>I’ll report back later on our progress.</p></section>Using AsciiDoc & Asciidoctor for blogging2021-01-12T11:31:39-08:002021-06-13T22:15:13-07:00Duncan Locktag:duncanlock.net,2021-01-12:/blog/2021/01/12/using-asciidoc-and-asciidoctor-for-blogging/<section aria-label="Preamble" id="preamble"><figure class="image-block"><img alt="Asciidoc" src="https://duncanlock.net/images/posts/using-asciidoc-and-asciidoctor-for-blogging/asciidoctor-logo-blueprint-cropped-compressed.svg" width="400"/>
<figcaption>Figure 1. Asciidoc.</figcaption></figure>
<p>I’ve been using reStructuredText for writing on this blog, because it has lots of built-in features that markdown doesn’t.</p>
<p>However, reStructuredText’s <em>actual syntax</em> is a bit… fiddly - particularly its non-atx headings, too many things relying on lining up white space, etc… If I don’t use it for a bit, I have to look up or copy <span class="amp">&</span> paste all the advanced syntax.</p>
<p>I’d prefer to use AsciiDoc, as it has all the extra features, and if you use Asciidoctor, <a href="https://docs.asciidoctor.org/asciidoc/latest/asciidoc-vs-markdown/">all the simple stuff is the same as markdown</a> - which isn’t (currently) standard AsciiDoc, but is a nice simplification.</p>
<p>The subset of features from reStructuredText (or Asciidoc) that Markdown doesn’t have – and that I’m <em>actually using</em> on this blog, are:</p>
<div class="ulist"><ul><li>Figure/Images with captions</li><li>Admonitions</li><li>Front-matter/Metadata</li><li>Footnotes</li></ul></div></section><section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/using-asciidoc-and-asciidoctor-for-blogging/asciidoctor-logo-blueprint-cropped-compressed.svg" alt="Asciidoc" width="400">
<figcaption>Figure 1. Asciidoc.</figcaption></figure>
<p>I’ve been using reStructuredText for writing on this blog, because it has lots of built-in features that markdown doesn’t.</p>
<p>However, reStructuredText’s <em>actual syntax</em> is a bit…​ fiddly - particularly its non-atx headings, too many things relying on lining up white space, etc…​ If I don’t use it for a bit, I have to look up or copy <span class="amp">&</span> paste all the advanced syntax.</p>
<p>I’d prefer to use AsciiDoc, as it has all the extra features, and if you use Asciidoctor, <a href="https://docs.asciidoctor.org/asciidoc/latest/asciidoc-vs-markdown/">all the simple stuff is the same as markdown</a> - which isn’t (currently) standard AsciiDoc, but is a nice simplification.</p>
<p>The subset of features from reStructuredText (or Asciidoc) that Markdown doesn’t have – and that I’m <em>actually using</em> on this blog, are:</p>
<div class="ulist"><ul><li>Figure/Images with captions</li><li>Admonitions</li><li>Front-matter/Metadata</li><li>Footnotes</li></ul></div></section>
<section class="doc-section level-1"><h2 id="_worse_is_better">Worse is better?</h2><p>Markdown has <em>lots</em> of problems. Most of these problems stem from two things: <a href="https://commonmark.org/">Until CommonMark</a> it was lazily, pointlessly, stubbornly non-standard - and it has very few structural or semantic features. Because of people wanting to add some of these features - and because there was no standard and therefore no way to extend one, Markdown has exploded into many fragmented semi-compatible dialects.</p>
<p>However, <a href="https://en.wikipedia.org/wiki/Worse_is_better">worse is better</a>. Markdown’s simplicity makes it pretty simple to implement, to the extent that there are native implementations of Markdown in probably every language: <a href="https://bitbucket.org/yiyus/md2html.awk"><span class="caps">AWK</span></a>, <a href="https://github.com/chadbraunduin/markdown.bash">Bash</a>, <a href="https://github.com/commonmark/cmark">C</a> …​ to <a href="https://github.com/kivikakk/koino">Zig</a>. There are <a href="https://github.com/markdown/markdown.github.com/wiki/Implementations">hundreds</a> of more-or-less complete <a href="https://github.com/search?q=markdown+implementation">markdown implementations</a> to choose from, to fit any project; here are <a href="https://github.com/commonmark/commonmark-spec/wiki/List-of-CommonMark-Implementations">fifty high quality maintained ones, that support CommonMark, sorted by language</a>, for example.</p>
<p>AsciiDoc is <em>much</em> better than Markdown, but these extra features for the <em>writer</em> come with extra complexity for the person <em>implementing</em> the tools. Probably as a consequence, the AsciiDoc ecosystem and tooling is very anemic and leaves <em>a lot</em> to be desired.</p>
<p>The <em>only</em> complete and well maintained AsciiDoc processor is <a href="https://asciidoctor.org/">Asciidoctor</a> - which is written in Ruby. That’s it - there are no other options as of early 2021<a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>. I don’t have anything against Ruby, particularly, but I don’t know it or use it for anything, so it’s not familiar. It <em>also</em> has all the same well know problems as Python with packaging/running projects <span class="amp">&</span> managing dependencies. Instead of pip/setuptools/virtualenv, etc…​ its gem/rbvenv/rake/bundler/rvm, etc…​ hundreds of global gem files, the whole giant mess. I get the impression that the Ruby version of this mess is…​ less of a problem somehow than the python one, but it’s still a mess.</p>
<p>So, the AsciiDoc story is obviously much more limited than the markdown story. If you want native markdown support, you got it, no matter what you’re doing. If you want native AsciiDoc support, you can only have it if your project is in Ruby (or Java <span class="amp">&</span> JavaScript with some caveats <a class="footnote-ref" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>). Other than that, you have to shell out and run a Ruby process - and have all the required Ruby dependencies installed.</p>
<p>This turns your document processing into a slower-than-it-could-be, annoying to setup and maintain, external black box. Don’t take this the wrong way - the contents of that black box are <em>fantastic</em> and good people have worked hard on them, but there are only a few of those people and pitting them against the markdown community - which is much, much larger - isn’t really fair.</p></section>
<section class="doc-section level-1"><h2 id="_what_are_the_consequences_of_this_for_blogging">What are the consequences of this for blogging?</h2><aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>AsciiDoc supports indexes, includes and all sorts of other complex documentation structure. I’m not going to go into those here - I’m just thinking about using an existing blogging system, with some/all posts written in AsciiDoc. If you want to get AsciiDoc to do <em>everything</em>, you can, but that’s outside the scope of this post. If you want that, you could check out <a href="https://antora.org/">Antora</a>, which is an AsciiDoc publishing tool written by the Asciidoctor team.</p></aside>
<section class="doc-section level-2"><h3 id="_editing_experience_isnt_as_good">Editing experience isn’t as good</h3><p>Most modern editors support AsciiDoc syntax highlighting and sometimes preview, via plugins. But, as expected, these aren’t as well-developed, numerous or as fully featured as the Markdown equivalents. If you’re using the Asciidoctor markdown-a-like syntax for the simple stuff, then you can just tell the editor it’s Markdown, and use the more fully featured markdown support.</p></section>
<section class="doc-section level-2"><h3 id="_not_much_native_support">Not much native support</h3><p>If you’re using a Ruby blog engine, it <em>might</em> have <a href="https://gist.github.com/briandominick/e5754cc8438dd9503d936ef65fffbb2d">native AsciiDoc support</a>, or your blog engine might have a plugin that supports it. Don’t expect every blog engine to support it, though. Some blog engines have an escape hatch that let you use <a href="https://pandoc.org/index.html">Pandoc</a> or some other custom command to process your content - and while Pandoc <em>does</em> have AsciiDoc support, it’s neither complete nor flawless.</p></section>
<section class="doc-section level-2"><h3 id="_does_native_support_matter">Does native support matter?</h3><p>Does the fact that the document processing is shelled out to a black box actually affect using AsciiDoc for blogging? Well, it makes publishing a bit slower to <em>a lot</em> slower, depending on the size of your site. But in most blog/publishing systems you have a template that defines the structure of the page with some kind of placeholder that says <code>{{content-goes-here}}</code>. It doesn’t really matter <em>how</em> the content is generated, as long as it is - and as long as the <span class="caps">HTML</span> that’s produced lines up with your <span class="caps">CSS</span> <span class="amp">&</span> is reasonable. Speaking of which…​</p></section>
<section class="doc-section level-2"><h3 id="_asciidoctors_built_in_html_output_could_be_better">Asciidoctor’s built-in <span class="caps">HTML</span> output could be better</h3><p>When <code>asciidoctor</code> processes your <code>.adoc</code> file you can tell it what converter (or “back-end”) you want to use - i.e. what kind of output you want. <a href="https://docs.asciidoctor.org/asciidoctor/latest/converters/">These are the built-in options</a>. This is what they say about the default <span class="caps">HTML</span> converter:</p>
<div class="quote-block"><blockquote><p>The <span class="caps">HTML</span> 5 converter (<code>html</code> or <code>html5</code>) generates <span class="caps">HTML</span> 5 styled with <span class="caps">CSS3</span>. This is the converter Asciidoctor uses by default.</p><footer>— <cite><a href="https://docs.asciidoctor.org/asciidoctor/latest/converters/#built-in-converters">Asciidoctor Doc: Built-in converters</a></cite></footer></blockquote></div>
<p>This is <em>technically</em> true, however, while the <span class="caps">HTML</span> it produces is <em>technically</em> <span class="caps">HTML5</span>, it’s also <code><div></code> soup. For example, this document processor for processing documents doesn’t output paragraph tags (<code><p>…​</p></code>) - it outputs this instead: <code><div class="paragraph"><p>…​</p></div></code> - and it does something like that for basically everything. To be fair, <a href="https://github.com/asciidoctor/asciidoctor/projects/1">they are aware of this <span class="amp">&</span> working on it</a>.</p>
<p>So, I’m not going to use that, I’ll just use some community written back-end/converter that’s better…​ oh…​ yeah. Well, luckily, in this case there actually <em>is</em> one: <a href="https://github.com/jirutka/asciidoctor-html5s">html5s</a> - which does a <em>much</em> better job.</p></section></section>
<section class="doc-section level-1"><h2 id="_asciidoc_rough_edges">AsciiDoc Rough Edges</h2><p>I’ve written a few articles from scratch and <a href="#_converting_your_existing_content_to_asciidoc">converted the existing 80 reStructuredText articles to AsciiDoc</a>, and it’s been <em>fairly</em> painless, but I <em>have</em> come across a few rough edges and problems with AsciiDoc.</p>
<section class="doc-section level-2"><h3 id="_footnotes">Footnotes</h3><p>AsciiDoc has <a href="https://docs.asciidoctor.org/asciidoc/latest/macros/footnote/">built-in support for footnotes</a>, but there are some rough edges:</p>
<div class="ulist"><ul><li><a href="https://github.com/asciidoctor/asciidoctor/issues/559">An improved footnote syntax would be nice</a>. It’s currently <span class="caps">OK</span>, but could be better.</li><li><a href="https://github.com/asciidoctor/asciidoctor/issues/3690#issuecomment-778956139">Footnotes that have no <span class="caps">ID</span> get duplicated</a>. So, this means that if you <a href="https://docs.asciidoctor.org/asciidoc/latest/macros/footnote/#externalizing-a-footnote">Externalize a footnote</a>, then refer to it more than once, it’ll get duplicated in the list of footnotes, unless you give it an <span class="caps">ID</span>. So, give footnotes IDs.</li><li><a href="https://github.com/asciidoctor/asciidoctor-pdf/issues/1397#issuecomment-780322751">Text formatting not being applied to footnotes</a> This means that text formatting, like <strong>bold</strong> and <code>monospace</code> don’t get processed in footnotes, they’ll come out literally, like *bold* or `monospace`. To fix this you need to use an “inline passthrough” which has slightly different syntax.</li></ul></div>
<p>The combination of these issues means that if you want externalized footnotes that work like the rest of your content, you have to give the footnote an <span class="caps">ID</span> and wrap the footnote definition in an inline pass-through. Because these become document attributes, you have to define them before you use them, so you should probably put these at the top.</p>
<p>This is more complex <span class="amp">&</span> convoluted than it needs to be - footnotes should just work. Anyway, it looks like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="asciidoc">:fn-disclaimer: pass:q[footnote:disclaimer[Opinions are *my own*.]]
A bold statement!{fn-disclaimer}
Another bold statement!{fn-disclaimer}</code></pre></div>
<p>They have a proposal for <a href="https://github.com/asciidoctor/asciidoctor/issues/559">an improved footnote syntax</a> - although it doesn’t talk about text formatting inside the footnote.</p></section>
<section class="doc-section level-2"><h3 id="_blockquotes">Blockquotes</h3><p>I took me ages poking around on GitHub before I found out how to set the link text in the citation for a quoted block. This is the basic syntax:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="asciidoc">[quote, attribution, citation title and information]
Quote or excerpt text</code></pre></div>
<p>You can put a <span class="caps">URL</span> in there, and it works, but giving the <span class="caps">URL</span> a title doesn’t seem to work. So this works:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="asciidoc">[quote, https://en.wikipedia.org/wiki/Main_Page]
Quote or excerpt text</code></pre></div>
<p>but this doesn’t:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="asciidoc">[quote, https://en.wikipedia.org/wiki/Main_Page[Wikipedia]]
Quote or excerpt text</code></pre></div>
<p>However, using the <a href="https://docs.asciidoctor.org/asciidoc/latest/blocks/blockquotes/#quoted-paragraph">Quoted paragraph</a> syntax works:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="asciidoc">"Quote or excerpt text"
-- https://en.wikipedia.org/wiki/Main_Page[Wikipedia]</code></pre></div>
<p><a href="https://github.com/asciidoctor/asciidoctor/issues/1254">Apparently, the correct way to do this with quoted blocks</a>, is to “use single quotes around the attribute value, that gives Asciidoctor the hint to apply normal substitutions (just like paragraph text)”<a class="footnote-ref" id="_footnoteref_2" href="#_footnote_2" title="View footnote 2" role="doc-noteref">[2]</a>. Not sure what that means at this point, but the docs on <a href="https://docs.asciidoctor.org/asciidoc/latest/subs/substitutions/">subtitutions are here</a>. This is what it looks like in this case:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="asciidoc">[quote, 'https://en.wikipedia.org/wiki/Main_Page[Wikipedia]']
Quote or excerpt text</code></pre></div></section></section>
<section class="doc-section level-1"><h2 id="_using_asciidoc_with_pelican">Using AsciiDoc with Pelican</h2><p>I’m currently using <a href="https://blog.getpelican.com/">Pelican</a> for this blog and writing this post in AsciiDoc. This is what you need to do to get that working.</p>
<p>First <a href="https://asciidoctor.org/#gem-install">install the Ruby dependencies <span class="amp">&</span> Asciidoctor itself</a>. Unlike me, you should listen to them and use <span class="caps">RVM</span> for this. Once you have that installed, you need to <a href="https://github.com/jirutka/asciidoctor-html5s#installation">install html5s and its dependencies</a>. Next, you need to add the <a href="https://github.com/getpelican/pelican-plugins/tree/master/asciidoc_reader">asciidoc_reader Pelican Plugin</a> and add it to your <code>pelicanconf.py</code></p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">PLUGINS</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">'asciidoc_reader'</span><span class="p">,</span>
<span class="p">]</span></code></pre></div>
<p>You should then set the Asciidoctor command line options. These will configure it to use the <code>html5s</code> backend and <a href="https://github.com/rouge-ruby/rouge">rouge for source code syntax highlighting</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">ASCIIDOC_OPTIONS</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">'-a source-highlighter=rouge'</span><span class="p">,</span>
<span class="s">'-a rouge-style=monokai'</span><span class="p">,</span>
<span class="s">'-r asciidoctor-html5s'</span><span class="p">,</span>
<span class="s">'-b html5s'</span>
<span class="p">]</span></code></pre></div>
<p>Rouge is compatible with pygments - which I was using previously and my theme is set up to expect, so this was a drop-in replacement - which is very convenient.</p>
<section class="doc-section level-2"><h3 id="_adding_removing_plugins">Adding <span class="amp">&</span> removing plugins</h3><p>The AsciiDoctor + htmls output has better figure output than reStructuredText + my Better Figures <span class="amp">&</span> Images Plugin, so I don’t need that anymore - provided that I convert all articles using figures to AsciiDoc. On the other hand, the <code>extract_toc</code> plugin doesn’t work for AsciiDoc + htmls output, so I copied it to a local plugin and modified it to work:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="c1"># -*- coding: utf-8 -*-
</span><span class="s">"""
Extract Table of Contents from AsciiDoc output from the htmls backend
========================
A Pelican plugin to extract table of contents (ToC) from `article.content` and
place it in its own `article.toc` variable for use in templates.
"""</span>
<span class="kn">from</span> <span class="nn">os</span> <span class="kn">import</span> <span class="n">path</span>
<span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span>
<span class="kn">from</span> <span class="nn">pelican</span> <span class="kn">import</span> <span class="n">signals</span><span class="p">,</span> <span class="n">readers</span><span class="p">,</span> <span class="n">contents</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">extract_asciidoc_toc</span><span class="p">(</span><span class="n">content</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="n">contents</span><span class="p">.</span><span class="n">Static</span><span class="p">):</span>
<span class="k">return</span>
<span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">content</span><span class="p">.</span><span class="n">_content</span><span class="p">,</span> <span class="s">"html.parser"</span><span class="p">)</span>
<span class="n">toc</span> <span class="o">=</span> <span class="bp">None</span>
<span class="n">toc</span> <span class="o">=</span> <span class="n">soup</span><span class="p">.</span><span class="n">find</span><span class="p">(</span><span class="s">"nav"</span><span class="p">,</span> <span class="nb">id</span><span class="o">=</span><span class="s">"toc"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">toc</span><span class="p">:</span>
<span class="n">toc</span><span class="p">.</span><span class="n">extract</span><span class="p">()</span>
<span class="n">content</span><span class="p">.</span><span class="n">_content</span> <span class="o">=</span> <span class="n">soup</span><span class="p">.</span><span class="n">decode</span><span class="p">()</span>
<span class="c1"># Remove: <h2 id="toc-title">Table of Contents</h2>
</span> <span class="n">toc</span><span class="p">.</span><span class="n">h2</span><span class="p">.</span><span class="n">decompose</span><span class="p">()</span>
<span class="c1"># Change all ordered lists to unordered
</span> <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="n">toc</span><span class="p">(</span><span class="s">"ol"</span><span class="p">):</span>
<span class="n">l</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="s">"ul"</span>
<span class="n">content</span><span class="p">.</span><span class="n">toc</span> <span class="o">=</span> <span class="n">toc</span><span class="p">.</span><span class="n">decode</span><span class="p">()</span>
<span class="n">logger</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="sa">f</span><span class="s">"ExtractAsciidocToc: content.toc: </span><span class="si">{</span><span class="n">content</span><span class="p">.</span><span class="n">toc</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">register</span><span class="p">():</span>
<span class="n">signals</span><span class="p">.</span><span class="n">content_object_init</span><span class="p">.</span><span class="n">connect</span><span class="p">(</span><span class="n">extract_asciidoc_toc</span><span class="p">)</span></code></pre></div></section>
<section class="doc-section level-2"><h3 id="_converting_your_existing_content_to_asciidoc">Converting your existing content to AsciiDoc</h3><p>This depends on the format of your existing content:</p>
<section class="doc-section level-3"><h4 id="_converting_restructuredtext_to_asciidoc">Converting reStructuredText to AsciiDoc</h4><p>I tried <em>lots</em> of different way of converting reStructuredText to AsciiDoc - and none of them are perfect.</p>
<p><a href="https://pandoc.org/">Pandoc</a> does a <em>reasonable</em> job, unless you use figures, which get pretty mangled. They are <a href="https://github.com/jgm/pandoc/projects/3#card-61138871">aware/working on this</a>. If you want to use pandoc, This is the basic command:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pandoc <span class="nt">--wrap</span><span class="o">=</span>preserve <span class="nt">-f</span> rst <span class="nt">-t</span> asciidoctor <span class="s2">"source.rst"</span> <span class="o">></span> <span class="s2">"dest.adoc"</span></code></pre></div>
<p>As well as figures, this <em>also</em> messes up metadata and pelican <code>{static}</code> links, so you probably want some pre- <span class="amp">&</span> post-processing to fix that. I wrote a little shell script to fix everything except the figures:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="shell"><span class="c">#</span>
<span class="c"># Pre-process</span>
<span class="c">#</span>
<span class="nb">cat</span> <span class="s2">"</span><span class="nv">$src_path</span><span class="s2">"</span> | <span class="se">\</span>
<span class="c"># Remove :alt: tags from figures & images, otherwise they get lost</span>
<span class="nb">sed</span> <span class="nt">-r</span> <span class="s1">'s/:alt: /\n/g'</span> | <span class="se">\</span>
<span class="c"># Tabs to spaces</span>
<span class="nb">sed</span> <span class="nt">-r</span> <span class="s1">'s/\t/ /g'</span> | <span class="se">\</span>
<span class="c">#</span>
<span class="c"># Convert rst to asciidoc using pandoc</span>
<span class="c">#</span>
pandoc <span class="nt">--wrap</span><span class="o">=</span>preserve <span class="nt">--from</span> rst <span class="nt">--to</span> asciidoctor | <span class="se">\</span>
<span class="c">#</span>
<span class="c"># Post-process</span>
<span class="c">#</span>
<span class="c"># Fix metadata syntax, from date:: to :date:</span>
<span class="nb">sed</span> <span class="nt">-r</span> <span class="s1">'N; s/^(.*)::\n /:\1:/g; P; D'</span> | <span class="se">\</span>
<span class="c"># Remove extra breaks created from figure caption conversion</span>
<span class="nb">sed</span> <span class="nt">-r</span> <span class="s1">'N; s/____\n//g; P; D'</span> | <span class="se">\</span>
<span class="c"># Fix Pelican {static} links</span>
<span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/%7B/{/g'</span> <span class="nt">-e</span> <span class="s1">'s/%7D/}/g'</span> <span class="se">\</span>
<span class="o">></span> <span class="s2">"</span><span class="nv">$src_folder</span><span class="s2">/</span><span class="nv">$src_name</span><span class="s2">"</span>.adoc</code></pre></div></section>
<section class="doc-section level-3"><h4 id="_fine_ill_write_my_own_converter">Fine, I’ll write my own converter…​</h4><p>I use figures quite a bit, so this wasn’t a very satisfactory solution. Thinking about it, reStructuredText is basically <em>the</em> Python documentation format, so I looked for <a href="https://docutils.sourceforge.io/"><code>docutils</code></a> based tools to convert reStructuredText to other things. I eventually found <a href="https://github.com/wsidl/sphinx_asciidoc">sphinx_asciidoc</a>, which sort of worked - and was a fairly straightforward python script that I could improve. I <a href="https://github.com/dflock/sphinx_asciidoc">forked it here</a> and fixed all the issues I found - <a href="https://github.com/dflock/sphinx_asciidoc/commits/master">fixing metadata, figures, tables, linked images and various other things</a>.</p>
<p>I developed <span class="amp">&</span> tested this by converting all 80 odd rst articles on this site to AsciiDoc and fixing all the issues that I fond in the converter.</p>
<p>Until pandoc fixes their figures, as far as I know, this is probably the best, highest fidelity way to convert reStructuredText to AsciiDoc. If you want to use this, do something like this to get it setup:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>git clone https://github.com/dflock/sphinx_asciidoc.git
<span class="gp">$</span><span class="w"> </span><span class="nb">cd </span>sphinx_asciidoc
<span class="gp">$</span><span class="w"> </span>python3 <span class="nt">-m</span> venv ~/venv/sphinx_asciidoc
<span class="gp">$</span><span class="w"> </span><span class="nb">source</span> ~/venv/sphinx_asciidoc/bin/activate
<span class="gp">$</span><span class="w"> </span>python3 <span class="nt">-m</span> pip <span class="nb">install</span> <span class="nt">-r</span> requirements.txt</code></pre></div>
<p>then this to run it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>python3 ./sphinx_asciidoc/writer.py source.rst</code></pre></div>
<p>This will create a <code>source.rst.adoc</code> file in the same folder. I tried to keep this as general purpose as possible, but there are <em>probably</em> some things in here which are specific to my documents. There is a section at the top of <code>writer.py</code> with some knobs to twiddle:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="c1">#
# Things that should be options, but aren't
#
# Output the rendered TOC from docutils, or just `:toc:`
</span><span class="bp">self</span><span class="p">.</span><span class="n">outputTOC</span> <span class="o">=</span> <span class="bp">False</span>
<span class="c1"># Table column alignment, if not specified. Can be <>^ or
# '' for unspecified.
</span><span class="bp">self</span><span class="p">.</span><span class="n">defaultTableColAlign</span> <span class="o">=</span> <span class="s">""</span>
<span class="c1"># Specify percentages for columns widths, or leave browser to auto-layout?
</span><span class="bp">self</span><span class="p">.</span><span class="n">defaultTableColWidths</span> <span class="o">=</span> <span class="bp">True</span>
<span class="c1"># Do you want to output the [1] ref's after the {footnote}, or let asciidoctor do it?
</span><span class="bp">self</span><span class="p">.</span><span class="n">outputFootnoteRef</span> <span class="o">=</span> <span class="bp">False</span></code></pre></div></section>
<section class="doc-section level-3"><h4 id="_converting_markdown_to_asciidoc">Converting Markdown to AsciiDoc</h4><p>If your content is in Markdown, you need <a href="https://github.com/asciidoctor/kramdown-asciidoc">Kramdown</a>. Kramdown is a very good markdown to AsciiDoc converter, that works great and produces flawless AsciiDoc - unsurprising, given that it’s written by <a href="https://github.com/mojavelinux">Dan Allen</a>, the same guy who largely runs the Asciidoctor project. Once you have Kramdown installed, you can just do: <code>$ kramdoc source.md</code> and it’ll create a <code>source.adoc</code> file in the same folder.</p></section></section></section>
<section class="doc-section level-1"><h2 id="_future_of_asciidoc_asciidoctor">Future of AsciiDoc <span class="amp">&</span> Asciidoctor</h2><p>There are a few promising projects that will help improve the AsciiDoc ecosystem.</p>
<section class="doc-section level-2"><h3 id="_the_asciidoc_specification">The AsciiDoc Specification</h3><p>The first and biggest one is that AsciiDoc is <a href="https://asciidoctor.org/news/2019/01/07/asciidoc-spec-proposal/">finally getting a proper spec</a>, under the umbrella of the Eclipse Foundation. This is something that Markdown never had until CommonMark - and that AsciiDoc has lacked up to now. What this means is:</p>
<div class="quote-block"><blockquote><p>The specification for the AsciiDoc language will include an open source specification document, which defines required and optional <span class="caps">API</span> definitions, semantic behaviours, data formats, and protocols, as well as an open source Technology Compatibility Kit (<span class="caps">TCK</span>) that developers can use to develop and test compatible implementations. …​ A compatible implementation, as defined by the <span class="caps">EFSP</span>, must fully implement all non-optional elements of a specification version, must fulfill all the requirements of the corresponding <span class="caps">TCK</span>, and must not alter the specified <span class="caps">API</span>.</p>
<p>For users and developers alike, the AsciiDoc specification will mean a clear, working definition of what AsciiDoc is and how it should be interpreted. Developers will be able to build implementations, tools, and services around AsciiDoc without risk of diluting its meaning or splintering it. In turn, users will have more options, greater document portability, and the assurance that compatible implementations and tools will handle their AsciiDoc documents according to a versioned specification.</p><footer>— <cite><a href="https://asciidoctor.org/news/2019/01/07/asciidoc-spec-proposal/">AsciiDoc Spec Proposal</a></cite></footer></blockquote></div>
<p>Here is the <a href="https://projects.eclipse.org/proposals/asciidoc-language">AsciiDoc Language project proposal</a> and the <a href="https://gitlab.eclipse.org/eclipse/asciidoc/asciidoc-lang/-/blob/main/process/scope.adoc">approved scope of the project</a>.</p>
<p>So, this should help prevent the fragmentation that plagues the Markdown ecosystem, as well a making it easier for people to develop AsciiDoc parsers <span class="amp">&</span> tools. Still nowhere near as easy as implementing a Markdown one, though - AsciiDoc is just more complex.</p>
<p>Having said that, this is a big project and most of the activity is taking place on <a href="https://www.eclipse.org/lists/asciidoc-wg">mailing lists</a> - there also now a <a href="https://asciidoc-wg.eclipse.org/committees/">website for the Working Group</a> which currently includes meeting minutes etc…​ There is now an AsciiDoc language repo for discussing the spec work, but it’s still early days: <a class="bare" href="https://gitlab.eclipse.org/eclipse/asciidoc/asciidoc-lang/">https://gitlab.eclipse.org/eclipse/asciidoc/asciidoc-lang/</a></p></section>
<section class="doc-section level-2"><h3 id="_libasciidoc">libasciidoc</h3><p><a href="https://github.com/bytesparadise/libasciidoc">Libsciidoc is a Golang library for processing AsciiDoc files</a>. This uses a <a href="https://en.wikipedia.org/wiki/Parsing_expression_grammar"><span class="caps">PEG</span> parser</a> with a formal grammar for AsciiDoc. It already supports a useful subset of AsciiDoc and is being slowly worked on by a few people, I think with the intention to use it with <a href="https://gohugo.io/">Hugo</a>, which will make a nice combination, when it’s done.</p>
<p>Like most software written in Go, it’s statically linked, which means no dependencies at all - you just need to put the <code>libacsiidoc</code> binary somewhere and run it. This is <em>really nice</em> compared to setting up and maintaining the Ruby dependencies required for Asciidoctor, or the <span class="caps">JS</span> <span class="amp">&</span> Java ones for Asciidoctor-J/Java, for example.</p>
<hr></section>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References:</h3></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">Asciidoctor can also be run on the <span class="caps">JVM</span> - Asciidoctor-j (Java) or in a Browser/Nodejs - Asciidoctor-js (JavaScript). These are both just the Ruby version running in different places - either using JRuby to run on the <span class="caps">JVM</span>, or using the Opal Ruby to JavaScript source-to-source compiler to run the Ruby code on a JavaScript <span class="caps">VM</span>. The Opal runtime + the AsciiDoc source weighs in at about 1.2Mb of <span class="caps">JS</span>. These are both a bit fat and slow and don’t <em>really</em> solve any of the AsciiDoc ecosystem’s problems. <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">Using single quotes doesn’t fix the formatting on footnotes, so I guess “normal substitutions” are different somehow? <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>Speedrunning Computer Games History with a 6yr Old - Part 12020-12-27T23:13:51-07:002021-06-14T06:46:02-07:00Duncan Locktag:duncanlock.net,2020-12-27:/blog/2020/12/27/speedrunning-computer-games-history-with-a-6yr-old-part-1/<section aria-label="Preamble" id="preamble"><figure class="image-block"><img alt="Galaxians vector" src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-1/galaxians-blueprint-crop-compressed.svg" width="500"/>
<figcaption>Figure 1. Galaxians.</figcaption></figure>
<p>As a family gift for Christmas 2020, I set up a <a href="https://www.raspberrypi.org/products/raspberry-pi-400/">Raspberry Pi 400</a> with <a href="https://retropie.org.uk/">Retropie</a>. The plan is to load a selection of the “best of the best” games from computer game history, starting with games up to, roughly, 1985.</p>
<p>The kid doesn’t really know any better, so we can play Space Invaders unsullied by time and expectations and enjoy a speedrun through gaming history, playing <em>just</em> the highlights.</p>
<p>I’ll add more games as we go along, progressing through gaming history, one classic game at a time.</p></section><section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-1/galaxians-blueprint-crop-compressed.svg" alt="Galaxians vector" width="500">
<figcaption>Figure 1. Galaxians.</figcaption></figure>
<p>As a family gift for Christmas 2020, I set up a <a href="https://www.raspberrypi.org/products/raspberry-pi-400/">Raspberry Pi 400</a> with <a href="https://retropie.org.uk/">Retropie</a>. The plan is to load a selection of the “best of the best” games from computer game history, starting with games up to, roughly, 1985.</p>
<p>The kid doesn’t really know any better, so we can play Space Invaders unsullied by time and expectations and enjoy a speedrun through gaming history, playing <em>just</em> the highlights.</p>
<p>I’ll add more games as we go along, progressing through gaming history, one classic game at a time.</p>
<aside class="admonition-block note" role="note"><h6 class="block-title"><span class="title-label">Note: </span>Screen time</h6><p>I’m going to mention screen time up-front. The kids get very little screen time Monday to Friday - partly because we want it this way and partly because there just isn’t time.</p>
<p>In the mornings, <em>if they get ready early</em>, they get the rest of the time before we leave, and the bedtime glide path starts with dinner at about 6pm - and no screens after dinner is a hard rule. They generally <em>don’t</em> get ready early, so they generally get about 0-30 mins <em>total</em> screen time <em>per week</em>, Monday-Friday.</p>
<p>Weekends…​ we have a lie-in and they watch cartoons until we get up, so usually an hour or two.</p></aside></section>
<section class="doc-section level-1"><h2 id="_why">Why?</h2><figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-1/Hard_reset_BBC_Micro_32K_Acorn_DFS-crop.webp" alt="A black screen, with the BBC Basic prompt and flashing cursor.">
<figcaption>Figure 2. The Mighty <span class="caps">BBC</span> <span class="caps">BASIC</span> prompt. This is what you saw when you booted it up - straight into an interactive programming environment: a 1980s <span class="caps">REPL</span>. Just type in <span class="caps">BASIC</span> code/commands and press Enter.</figcaption></figure>
<p>When I was around 8 yrs old, my parents bought a <a href="https://en.wikipedia.org/wiki/BBC_Micro"><span class="caps">BBC</span> Micro Model B</a> - an early 8-bit home computer. Home computers were very new back then and I’d never used one before. We played games on it. A lot. <a href="http://bbcmicro.co.uk/game.php?id=3525">Asteriods</a>, <a href="http://bbcmicro.co.uk/game.php?id=1934">Frogger</a>, <a href="http://bbcmicro.co.uk/game.php?id=353">Missile Strike</a>, <a href="http://bbcmicro.co.uk/game.php?id=89">Twin Kingdom Valley</a>, <a href="http://bbcmicro.co.uk/game.php?id=2647">Racer</a>, <a href="http://bbcmicro.co.uk/index.php?search=Repton&on_Z=on">Repton</a>, <a href="http://bbcmicro.co.uk/game.php?id=25">Chuckie Egg</a>, <a href="https://en.wikipedia.org/wiki/Elite_(video_game)">Elite</a>, etc, etc…​ all loaded (very slowly) from <a href="https://en.wikipedia.org/wiki/Cassette_tape">cassette tape</a>. There were also lots of games and other programs that came as code, printed out in books or magazines, which you then typed in.</p>
<p>Later, I started writing my own little <span class="caps">BASIC</span> programs. I think partly because that was what Mum <span class="amp">&</span> Dad actually <em>wanted</em> us to do with this rather expensive computer, and partly because it was <em>interesting</em> to get the computer to do things - and gratifying when they <em>finally</em> worked.</p>
<p>Like many others, this little machine changed the trajectory of my life. I’ve been fortunate to have made a reasonable living out of programming computers. It’s <em>still</em> interesting to get computers to do things that you thought up - and it’s <em>still</em> gratifying when it <em>finally</em> works, even after all this time.</p>
<p>I have no illusions that we can re-create this experience - you can’t step into the same river twice. However, I hope that we might be able to capture a <em>few</em> of the best bits, the most fun <span class="amp">&</span> most useful, while making new memories of our own.</p>
<p>I think that the 6yr old is currently in a narrow window where this might be possible. They haven’t really had much exposure to computer games, and the <span class="caps">COVID</span>-19 pandemic will keep them out of their peers houses for a while longer. But, because kids these days are surrounded by advanced computers, with modern 3D graphics and sound, I can’t leave it until they’re 8 or 9 - the window will have closed and modernity will have forever changed expectations.</p>
<p>I think that I’ve got about a year or so, at the most, before Minecraft and Roblox sweep this all away and close the window. I’m going to see if we can get to 1992 before then - it’s around this time that I think the games will start to stand up on their own, graphics <span class="amp">&</span> fun wise, against Minecraft. Maybe? Alright, probably not. We’ll see how it goes - either way, it’ll be fun while it lasts.</p>
<p>It’s also around that point that the Raspberry Pi 400 will start to struggle to emulate the more demanding games <span class="amp">&</span> systems, so we’ll have to do something else then anyway.</p>
<section class="doc-section level-2"><h3 id="_what_do_you_have_against_minecraftroblox">What do you have against Minecraft/Roblox?</h3><p>Absolutely nothing, I think they’re great! The problem is, once you play modern games with modern(-ish) 3d graphics, your expectations are forever changed. It makes it harder to appreciate games which don’t have similarly modern graphics. This isn’t a fault of older games, it’s just a change in expectations - it’s hard to put that toothpaste back into the tube.
When we get there, I hope we’ll play Minecraft <span class="amp">&</span> Roblox together too - and maybe <a href="https://www.codeadvantage.org/coding-for-kids-blog/minecraft-vs-roblox">use them to introduce coding</a>.</p></section></section>
<section class="doc-section level-1"><h2 id="_a_modern_home_computer">A Modern “Home Computer”</h2><p>I’m intending the Raspberry Pi 400 to be “the family/kids/home computer” for a while:</p>
<div class="quote-block"><blockquote><p>Featuring a quad-core 64-bit processor, <span class="caps">4GB</span> of <span class="caps">RAM</span>, wireless networking, dual-display output, and 4K video playback, as well as a 40-pin <span class="caps">GPIO</span> header, Raspberry Pi 400 is a powerful, easy-to-use computer built into a neat and portable keyboard.</p><footer>— <cite><a class="bare" href="https://www.raspberrypi.org/products/raspberry-pi-400/">https://www.raspberrypi.org/products/raspberry-pi-400/</a></cite></footer></blockquote></div>
<p>We can use it to play games on - but it’s also a regular Linux box, running Raspbian (a Debian distro) and can run a huge array of Linux software. It’s easy to change its personality but just booting it off a different <span class="caps">USB</span> stick/<span class="caps">SD</span> card, with a different <span class="caps">OS</span> <span class="amp">&</span> apps on. I’m intending to use it to introduce the kids to programming, as well as basic electronics and robotics - all of which are well-supported by the Pi’s hardware and huge maker community.</p></section>
<section class="doc-section level-1"><h2 id="_hardware">Hardware</h2><p>We live in a pretty small apartment, so everything needed to be able to pack away as small as possible, without breaking the bank. The combination of the Raspberry Pi 400, which released at just the right time for this project, and a portable monitor - means that the system packs away fairly neatly.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-1/IMG_20210331_155217-small.jpg" alt="The assembled system, with Raspberry Pi, monitor and controllers, sitting on an Ikea Lack table. Underneath the table are two boxes and a small pouch; the system packs away into these when not in use.">
<figcaption>Figure 3. The assembled system, with Raspberry Pi, monitor and controllers, sitting on an Ikea Lack table. Underneath the table are two boxes and a small pouch; the system packs away into these when not in use.</figcaption></figure>
<div class="ulist"><ul><li>The system itself is a <a href="https://www.raspberrypi.org/products/raspberry-pi-400/">Raspberry Pi 400</a></li><li>The monitor is a <a href="https://amzn.to/3fxx5BN">Portable 15.6” <span class="caps">TN</span> Screen Full <span class="caps">HD</span> 1920×1080</a></li><li>The controllers are some retro <span class="caps">NES</span> style <span class="caps">USB</span> ones. I actually bought two pairs, <a href="https://amzn.to/2Ppq2jR">these ones</a> and some Buffalo ones from this <a href="https://www.ebay.ca/usr/ship_japan">awesome seller on eBay</a>.</li></ul></div>
<p>The monitor arrived without the bent metal wire stand, so I ended up making some decorative stands out of cardboard and covering them with retro game artwork:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/speedrunning-computer-games-history-with-a-6yr-old-part-1/IMG_20210331_155229-small.jpg" alt="The monitor stands I made from cardboard.">
<figcaption>Figure 4. These stands came out pretty well, if I do say so myself - and they’ve unexpectedly been a hit with the kid, too. I often get “Which game is this?” or “I want to play this one” - pointing at one of the pictures on these stands.</figcaption></figure></section>
<section class="doc-section level-1"><h2 id="_software">Software</h2><p>It’s running <a href="https://retropie.org.uk/">Retropie</a>, which is a nice retro gaming system, which makes installing and configuring everything pretty simple:</p>
<div class="quote-block"><blockquote><p>[Retropie] builds upon Raspbian, EmulationStation, RetroArch and many other projects to enable you to play your favourite Arcade, home-console, and classic <span class="caps">PC</span> games with the minimum set-up. For power users it also provides a large variety of configuration tools to customise the system as you want.</p><footer>— <cite><a href="https://retropie.org.uk/">Retropie</a></cite></footer></blockquote></div>
<p>I used the Retropie image to install this, which means you just need to write the image to an <span class="caps">SD</span> card and boot it up. You can use <a href="https://www.balena.io/etcher/">Balena Etcher</a>, the <a href="https://www.raspberrypi.org/blog/raspberry-pi-imager-imaging-utility/">Raspberry Pi Imager</a>, or <a href="https://www.raspberrypi.org/documentation/installation/installing-images/linux.md">dd</a> to do this. <a href="https://retropie.org.uk/docs/First-Installation/">Installation instructions are here</a>.</p>
<p>The games were selected from my memory, and various lists:</p>
<div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/List_of_video_games_considered_the_best">Wikipedia: List of video games considered the best</a></li><li><a href="https://www.denofgeek.com/games/the-15-greatest-zx-spectrum-games-ever-made/">The 15 Greatest <span class="caps">ZX</span> Spectrum Games Ever Made</a></li><li><a href="https://www.retrogamer.net/top_10/top-ten-commodore-64-games/">Top Ten Commodore 64 Games</a></li><li><a href="https://www.lemon64.com/games/votes_list.php">The Lemoners Top Commodore 64 Games</a></li><li><a href="https://www.retrogamer.net/top_10/top-ten-apple-ii-games/">Top Ten Apple <span class="caps">II</span> Games</a></li></ul></div>
<p>We started with the following emulated systems <span class="amp">&</span> games:</p>
<section class="doc-section level-2"><h3 id="_arcade">Arcade</h3><p>These all work flawlessly in <code>lr-mame</code>, with perfect controller mappings out of the box.</p>
<div class="ulist three-columns"><ul><li><a href="https://en.wikipedia.org/wiki/Galaga_%2788">Galaga ‘88</a> (1988)</li><li><a href="https://en.wikipedia.org/wiki/Galaxian">Galaxian</a> (1979)</li><li><a href="https://en.wikipedia.org/wiki/Pac-Man">Pac-Man</a> (1980)</li><li><a href="https://en.wikipedia.org/wiki/Frogger">Frogger</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Qix">Qix</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Defender_(1981_video_game)">Defender</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Q*bert">Qbert</a> (1982)</li><li><a href="https://en.wikipedia.org/wiki/Donkey_Kong_(video_game)">Donkey Kong</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Centipede_(video_game)">Centipede</a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/BurgerTime">BurgerTime</a> (1982)</li><li><a href="https://en.wikipedia.org/wiki/Arkanoid">Arkanoid</a> (1986)</li></ul></div></section>
<section class="doc-section level-2"><h3 id="_zx_spectrum"><span class="caps">ZX</span> Spectrum</h3><p>The <a href="https://github.com/chernandezba/zesarux">Zesarux emulator</a> is a pretty great, but the speccy graphics are a bit basic, and they’re mostly intended for keyboard play, so we have to remap them for the controllers.</p>
<div class="ulist three-columns"><ul><li><a href="https://en.wikipedia.org/wiki/Bubble_Bobble">Bubble Bobble</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Tetris">Tetris</a> (1988)</li><li><a href="https://en.wikipedia.org/wiki/R-Type">R-Type</a> (1988)</li><li><a href="https://en.wikipedia.org/wiki/Manic_Miner">Manic Miner</a> (1983)</li><li><a href="https://en.wikipedia.org/wiki/Knight_Lore">Knight Lore</a> (1984)</li><li><a href="https://en.wikipedia.org/wiki/Jet_Set_Willy">Jet Set Willy</a> (1984)</li><li><a href="https://en.wikipedia.org/wiki/Head_over_Heels_(video_game)">Head Over Heels</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Dizzy_(series)">Dizzy</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Chuckie_Egg">Chuckie Egg</a> (1983)</li><li><a href="https://en.wikipedia.org/wiki/Chase_H.Q.">Chase <span class="caps">H.Q.</span></a> (1989)</li><li><a href="https://en.wikipedia.org/wiki/Chaos:_The_Battle_of_Wizards">Chaos - The Battle of the Wizards</a> (1985)</li><li><a href="https://en.wikipedia.org/wiki/Atic_Atac">Atic Atac</a> (1983)</li><li><a href="https://en.wikipedia.org/wiki/Deathchase">Deathchase</a> (1983)</li></ul></div></section>
<section class="doc-section level-2"><h3 id="_atari_2600">Atari 2600</h3><div class="ulist"><ul><li><a href="https://en.wikipedia.org/wiki/Pitfall!">Pitfall!</a> (1982)</li></ul></div></section>
<section class="doc-section level-2"><h3 id="_apple_ii">Apple <span class="caps">II</span></h3><p>This is a keyboard oriented emulator, so controller mapping, etc…​ Also, the sound is pretty bleepy.</p>
<div class="ulist three-columns"><ul><li><a href="https://en.wikipedia.org/wiki/Lode_Runner">Lode Runner</a> (1983)</li><li><a href="https://en.wikipedia.org/wiki/Choplifter">Choplifter</a> (1982)</li><li><a href="https://en.wikipedia.org/wiki/Cavern_Creatures">Cavern Creatures</a> (1983)</li><li><a href="https://en.wikipedia.org/wiki/Where_in_the_World_Is_Carmen_Sandiego%3F_(1985_video_game)">Where in the World is Carmen Sandeigo</a> (1985) - this has two floppy disks and it’s annoying to swap disks in the emulator. Also text heavy.</li></ul></div></section>
<section class="doc-section level-2"><h3 id="_commodore_64">Commodore 64</h3><p>This emulator doesn’t seem to work very well and the games are difficult to get working <span class="amp">&</span> figure out controls. Consequently, we really haven’t played any of these so far. I’m sure it’s just a matter of me sitting down and figuring out how to configure it properly.</p>
<div class="ulist three-columns"><ul><li><a href="https://en.wikipedia.org/wiki/Zak_McKracken_and_the_Alien_Mindbenders">Zak McKracken <span class="amp">&</span> the Alien Mindbenders</a> (1988)</li><li><a href="https://en.wikipedia.org/wiki/Wizball">Wizball</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Winter_Games">Winter Games</a> (1985)</li><li><a href="https://en.wikipedia.org/wiki/Turrican">Turrican 1</a> (1990)</li><li><a href="https://en.wikipedia.org/wiki/Turrican_II:_The_Final_Fight">Turrican 2: Final Fight</a> (1991)</li><li><a href="https://en.wikipedia.org/wiki/Turbo_Outrun">Turbo OutRun</a> (1989)</li><li><a href="https://www.c64-wiki.com/wiki/The_Sentinel">The Sentinel</a> (1986)</li><li><a href="https://en.wikipedia.org/wiki/M.U.L.E."><span class="caps">M.U.L.E.</span></a> (1983)</li><li><a href="https://en.wikipedia.org/wiki/Manic_Miner">Manic Miner</a> (1983)</li><li><a href="https://en.wikipedia.org/wiki/Maniac_Mansion">Maniac Mansion</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Lemmings_(video_game)">Lemmings</a> (1991)</li><li><a href="https://en.wikipedia.org/wiki/Laser_Squad">Laser Squad with Expansion Missions</a> (1988)</li><li><a href="https://en.wikipedia.org/wiki/International_Karate_%2B">International Karate +</a> (1987)</li><li><a href="https://en.wikipedia.org/wiki/Boulder_Dash">Boulder Dash</a> (1984)</li></ul></div>
<p>We should probably play the LucasArts games on <a href="https://www.scummvm.org/">ScummVM</a> anyway.</p></section>
<section class="doc-section level-2"><h3 id="_zmachine">ZMachine</h3><p>These are all text adventures, so require reading <span class="amp">&</span> writing.</p>
<div class="ulist three-columns"><ul><li><a href="https://en.wikipedia.org/wiki/Zork">Zork I</a> (1980)</li><li><a href="https://en.wikipedia.org/wiki/Zork_II">Zork <span class="caps">II</span></a> (1981)</li><li><a href="https://en.wikipedia.org/wiki/Zork_III">Zork <span class="caps">III</span></a> (1982)</li><li><a href="https://en.wikipedia.org/wiki/Starcross_(video_game)">Starcross</a> (1982)</li></ul></div></section></section>
<section class="doc-section level-1"><h2 id="_ill_report_back_later">I’ll report back later…​</h2><p>I realize that’s too many games, but…​ I got a bit carried away? Probably we won’t get around to playing all of these before we pass them by for newer pastures, but that’s life.</p>
<p>So, we’ll try it out and see how we get on – and I’ll report back later on our progress.</p></section>Fermented Bánh Mì/Đồ Chua(ish) Recipe2020-04-28T14:22:17-07:002021-07-11T00:00:37-07:00Duncan Locktag:duncanlock.net,2020-04-28:/blog/2020/04/28/fermented-banh-mi-do-chua-ish-recipe/<section id="preamble" aria-label="Preamble"><figure class="image-block small"><img src="https://duncanlock.net/images/posts/fermented-banh-mi-ish-recipe/IMG_20210424_213450-smaller.webp" alt="Spiralized carrots & daikon radish, with halved Jalapeno, in the brine ready to ferment.">
<figcaption>Figure 1. Spiralized carrots <span class="amp">&</span> daikon radish, with halved Jalapeno, in the brine ready to ferment.</figcaption></figure>
<p>This is a rather loose adaptation of a <a href="https://www.google.com/search?q=%C4%91%E1%BB%93+chua">traditional Vietnamese recipe</a>, often used as a filling in <a href="https://en.wikipedia.org/wiki/B%C3%A1nh_m%C3%AC">Bánh Mì</a>. You can google for lots of variations. Đồ Chua is often made as a vinegar pickle, with added sugar – <em>this</em> recipe is a keto/paleo <span class="amp">&</span> sugar-free lacto-fermented version.</p>
<p>The basic idea is to spiralize (or grate) <span class="amp">&</span> mix everything, except the chilies. Then pack all the spiralized vegetables into jars, with the halved chilies slid down the side, topped up with brine.</p>
<p>Start by making the basic brine and leave to dissolve while you prepare the vegetables.</p></section>
<section class="doc-section level-1 noclear"><h2 id="_basic_brine">Basic Brine</h2><p>Important to measure carefully. This is the brine I use for most simple ferments, unless I find a recipe that has a good reason for using something different …</p></section><section id="preamble" aria-label="Preamble"><figure class="image-block small"><img src="https://duncanlock.net/images/posts/fermented-banh-mi-ish-recipe/IMG_20210424_213450-smaller.webp" alt="Spiralized carrots & daikon radish, with halved Jalapeno, in the brine ready to ferment.">
<figcaption>Figure 1. Spiralized carrots <span class="amp">&</span> daikon radish, with halved Jalapeno, in the brine ready to ferment.</figcaption></figure>
<p>This is a rather loose adaptation of a <a href="https://www.google.com/search?q=%C4%91%E1%BB%93+chua">traditional Vietnamese recipe</a>, often used as a filling in <a href="https://en.wikipedia.org/wiki/B%C3%A1nh_m%C3%AC">Bánh Mì</a>. You can google for lots of variations. Đồ Chua is often made as a vinegar pickle, with added sugar – <em>this</em> recipe is a keto/paleo <span class="amp">&</span> sugar-free lacto-fermented version.</p>
<p>The basic idea is to spiralize (or grate) <span class="amp">&</span> mix everything, except the chilies. Then pack all the spiralized vegetables into jars, with the halved chilies slid down the side, topped up with brine.</p>
<p>Start by making the basic brine and leave to dissolve while you prepare the vegetables.</p></section>
<section class="doc-section level-1 noclear"><h2 id="_basic_brine">Basic Brine</h2><p>Important to measure carefully. This is the brine I use for most simple ferments, unless I find a recipe that has a good reason for using something different:</p>
<div class="ulist"><ul><li>45g salt per litre of water</li><li>Mix to dissolve</li></ul></div></section>
<section class="doc-section level-1"><h2 id="_ingredients">Ingredients</h2><p>I’ve only tried this with daikon radish – if you can’t get that, you could <em>probably</em> use whatever radish you can get, although making it with lots of little radishes might be a bit fiddly.</p>
<div class="ulist"><ul><li>1 Daikon Radish (650g)</li><li>6 carrots (650g)</li><li>85g ginger</li><li>1 jalapeño/green/red chili (to taste)</li><li>Basic Brine to cover</li></ul></div></section>
<section class="doc-section level-1"><h2 id="_instructions">Instructions</h2><figure class="image-block align-right small"><img src="https://duncanlock.net/images/posts/fermented-banh-mi-ish-recipe/IMG_20210423_235830-smaller.webp" alt="Spiralizing the carrots & radish.">
<figcaption>Figure 2. Spiralizing the carrots <span class="amp">&</span> radish.</figcaption></figure>
<p>Makes about two large (2l) mason jars worth, filling them about ⅔ full, leaving ⅓ for a large amount of headspace at the top.</p>
<div class="olist arabic"><ol class="arabic"><li>Lightly wash/rinse all the vegetables, but leave the skin on everything, including the ginger.</li><li>Grate or spiralize about equal amounts of daikon radish <span class="amp">&</span> carrots.</li><li>Grate the ginger with skin on.</li><li>Cut chili in half lengthwise, leaving the insides <span class="amp">&</span> seeds in.</li><li>If you spiralized the carrots <span class="amp">&</span> radish – which I would recommend, then roughly cut the pile with scissors, so strands are not too long.</li><li>Mix carrot, radish <span class="amp">&</span> ginger together in a large bowl.</li><li>Put a handful of the carrot <span class="amp">&</span> radish mixture in the bottom of a jar. Push chili halves down the sides, cut side facing outwards, so that the mixture holds them up.</li><li>Fill jar roughly ⅔ full with the mixture, packing it down as you go, to remove air pockets.</li></ol></div>
<section class="doc-section level-2"><h3 id="_headspace_cover">Headspace <span class="amp">&</span> Cover</h3><p>I usually leave this to ferment for a few months – a pretty long time. During this time, some water will evaporate, so the level of brine above the ferment will drop. This means that you need to start with a bit extra. Generally, the longer you leave a ferment running, the more likely it is that you will get some kind of yeast or mould growing on the top of the brine. These two things mean that to be successful, you need to start with a large amount of clear brine above the ferment – and be careful to “cap” the ferment with something to keep everything down, underneath the brine.</p>
<figure class="image-block small"><img src="https://duncanlock.net/images/posts/fermented-banh-mi-ish-recipe/IMG_20210210_210953-smaller.webp" alt="A cover or plug, made from a slice of rutabaga.">
<figcaption>Figure 3. A cover or plug, made from a slice of rutabaga.</figcaption></figure>
<p>I usually make a plug or cap by cutting a thin slice of rutabaga or swede, about 5 mm thick and about the same diameter as the widest part of the jar. You can then bend this to get it through the neck of the jar, then open it out and push it down on top of the carrot <span class="amp">&</span> radish mixture. Then put a weight on top of that, then top up the jar with the brine.</p>
<p>You could also use a few cabbage or collard leaves as a cap, instead of a rutabaga slice - you just need something that makes a good enough seal to stop things floating to the surface and will be okay sitting in brine for a few months.</p>
<figure class="image-block align-right small"><img src="https://duncanlock.net/images/posts/fermented-banh-mi-ish-recipe/IMG_20210424_212231-smaller.webp" alt="Lots of brine headspace at the top of the jar.">
<figcaption>Figure 4. Leave about ⅓ of the jar empty at the top, for the brine headspace, to protect the ferments below.</figcaption></figure>
<p>The fermentation is usually very active to start, then settles down. I usually leave for ~8-10 weeks, depending on temperature, etc…​ I like the flavours to mellow and combine over the long ferment time – but I would suggest that the first time you make them, you open them and taste them every two weeks or so and see what you like best.
After my ferment, the Radish is translucent it’s slightly soft/al dente, and the flavours have combined <span class="amp">&</span> mellowed.</p>
<p>If you forget about them for a few weeks (or months) …​ they’ll be fine!</p></section></section>Basic Pickles or Fermented Cucumbers Recipe2019-08-10T23:01:23-07:002021-06-16T22:52:15-07:00Duncan Locktag:duncanlock.net,2019-08-10:/blog/2019/08/10/basic-pickles-or-fermented-cucumbers-recipe/<section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/basic-pickles-or-fermented-cucumbers-recipe/IMG_20200229_201611-smaller.webp" alt="Two mason jars, full of fermenting cucumbers." width="400">
<figcaption>Figure 1. Two mason jars, full of cucumbers, just gone in. Both using this recipe, one with added chilli flakes, one with added Sichuan pepper.</figcaption></figure>
<p>Here’s my lacto-fermented cucumbers recipe. This is adapted from the great <a href="https://amzn.to/3vsrC3H">Wild Fermentation book</a>. I’ve honed this recipe to perfection over many trials – but you’ll need to try it a few times and adjust it to your particular cucumbers, climate <span class="amp">&</span> tastes, I expect.</p>
<p>Although mini-cucumbers are the archetypal “pickle”, this recipe is not limited to just cucumbers. Pretty much any crunchy vegetable can be fermented like this. Basically, any vegetable that’s crunchy will ferment well this way.</p>
<p>This is essentially two recipes: one for making the “basic brine” – which I use for lots of similar ferments – and one for making the fermented cucumbers, using the brine. Both recipes are very simple …</p></section><section id="preamble" aria-label="Preamble"><figure class="image-block"><img src="https://duncanlock.net/images/posts/basic-pickles-or-fermented-cucumbers-recipe/IMG_20200229_201611-smaller.webp" alt="Two mason jars, full of fermenting cucumbers." width="400">
<figcaption>Figure 1. Two mason jars, full of cucumbers, just gone in. Both using this recipe, one with added chilli flakes, one with added Sichuan pepper.</figcaption></figure>
<p>Here’s my lacto-fermented cucumbers recipe. This is adapted from the great <a href="https://amzn.to/3vsrC3H">Wild Fermentation book</a>. I’ve honed this recipe to perfection over many trials – but you’ll need to try it a few times and adjust it to your particular cucumbers, climate <span class="amp">&</span> tastes, I expect.</p>
<p>Although mini-cucumbers are the archetypal “pickle”, this recipe is not limited to just cucumbers. Pretty much any crunchy vegetable can be fermented like this. Basically, any vegetable that’s crunchy will ferment well this way.</p>
<p>This is essentially two recipes: one for making the “basic brine” – which I use for lots of similar ferments – and one for making the fermented cucumbers, using the brine. Both recipes are very simple.</p></section>
<section class="doc-section level-1"><h2 id="_basic_brine_recipe_per_litre">Basic Brine Recipe (per litre)</h2><div class="ulist"><ul><li>1 litre water</li><li>42.5g salt (just regular table salt, <em>not</em> lo-salt <span class="amp">&</span> no added ingredients)</li><li>¼ teaspoon <a href="https://amzn.to/2TbvOaZ">Calcium Chloride</a> (optional, but keeps things crunchy)</li></ul></div>
<p>Mix this all together in a clean mixing bowl or jug, until the salt and calcium chloride dissolve into the water.</p></section>
<section class="doc-section level-1"><h2 id="_basic_pickles_recipe">Basic Pickles Recipe</h2><section class="doc-section level-2"><h3 id="_equipment">Equipment</h3><div class="ulist"><ul><li>Clean wide mouth mason jars - or any vaguely similar container</li><li>Either a fermentation weight <span class="amp">&</span> lid, or a small ziplock bag 50% full of tap water, as a weight/seal for the container</li><li>A Cabbage leaf</li></ul></div></section>
<section class="doc-section level-2"><h3 id="_ingredients">Ingredients</h3><div class="ulist"><ul><li>Fresh un-waxed small cucumbers/gherkins - however many you want to make</li></ul></div>
<section class="doc-section level-3"><h4 id="_per_jar">Per jar:</h4><div class="ulist"><ul><li>2 sprigs fresh dill</li><li>1 teaspoon black peppercorns (you can use ground pepper if you don’t have whole)</li><li>1 teaspoon mustard or coriander seeds (optional)</li><li>A bay leaf (optional)</li></ul></div></section></section>
<section class="doc-section level-2"><h3 id="_instructions">Instructions</h3><div class="olist arabic"><ol class="arabic"><li>Soak cucumbers in cold water for a bit.</li><li>While the cucumbers are soaking, <a href="#_basic_brine_recipe_per_litre">make the brine solution</a>. Roughly 80% of the space in your jars will be filled with cucumbers, so don’t make <em>too much</em> brine.</li><li>Rinse the cucumbers, taking care to not bruise them. Don’t clean them with detergent or anything - just rinse any dirt off. The cucumber skin is covered in the good lactobacteria that will do the fermenting for us, so don’t clean them too much.</li><li>Top <span class="amp">&</span> tail the cucumbers, making sure their blossoms ends are removed.</li><li>Put dill, peppercorns <span class="amp">&</span> seeds into jars.</li><li>Pack cucumbers into the jars, length-ways, or cut sideways. Leave several inches of headspace at the top of the jar - enough to fit the ziplock bag or fermenting weight.</li><li>Pour brine over the cucumbers, put cabbage leaf over the top to stop anything floating to the surface. Again, leave several inches of clear brine at the top of the jar.</li><li>Depending on which you’re using, either put the weight <span class="amp">&</span> lid on – or the water filled ziplock bag into the top of the jar, as a seal <span class="amp">&</span> weight, to keep everything underneath the brine.</li><li>Put the jars somewhere with a steady temperature, away from direct sunlight, roughly room temperature. Warmer makes them go faster, too warm makes them into mush; room temp is fine.</li><li>After a few days the water should go cloudy - this means it’s working and they’ve started to ferment.</li><li>Check the jar every few days. Skim any mould/yeast from the surface of the water, but don’t worry if you can’t get it all. If there’s mold, be sure to rinse the bag <span class="amp">&</span> put back. Taste the pickles after about a week.</li><li>Feel free to remove and eat the pickles as they continue to ferment, until you decide how sour you like them – longer fermentation will increase sourness. Continue to check occasionally.</li><li>Eventually, after one to four weeks (depending on the temperature), the pickles will be fully sour. When they reach your preferred sourness, move them to the fridge to slow fermentation down to almost (but not quite) a standstill. I usually leave mine fermenting for roughly 2 weeks.</li></ol></div>
<p>Once you’ve done this a couple of times and got it working, feel free to experiment by adding extra stuff for different tastes. Here are a few ideas:</p>
<div class="ulist"><ul><li>Garlic cloves</li><li>Chili, either whole, sliced, or dried flakes</li><li>Sliced or grated Radish</li></ul></div>
<p>Good luck and let me know how you get on!</p></section></section>I finally figured out my mysterious 418/Unused HTTP Status Code2016-03-22T22:21:19-07:002021-06-07T22:21:48-07:00Duncan Locktag:duncanlock.net,2016-03-22:/blog/2016/03/22/finally-figured-out-my-mysterious-418unused-http-status-code-dreamhost/<figure class="image-block"><img src="https://duncanlock.net/images/posts/finally-figured-out-my-mysterious-418unused-http-code-dreamhost/teapot.png" alt="Blueprint style diagram showing a Teapot.">
<figcaption>Figure 1. A teapot, cut in half. Sort of. Original clipart Kitchen Utensils Silhouette, by <span class="caps">GDJ</span>, Public Domain. <a href="https://en.wikipedia.org/wiki/Teapot">More on Teapots</a>.</figcaption></figure>
<p>I’ve had a mysterious broken page on this site for a while - but been too busy to look into it. My <a href="https://duncanlock.net/blog/2013/08/27/comprehensive-linux-backups-with-etckeeper-backupninja/">Comprehensive Linux Backups with etckeeper <span class="amp">&</span> backupninja</a> article has been refusing to load, and returning a weird <span class="caps">HTTP</span> 418 Unused status code instead. I finally made the time to figure out the cause.</p>
<p>It turned out that this was being caused by the Apache/<span class="caps">PHP</span> <code>mod_security</code> module. This is a static website - there’s no <span class="caps">PHP</span> anywhere - so why would that be a problem? Well, so far I’ve been very happily hosting the site on my old DreamHost shared hosting account - which comes with Apache <span class="amp">&</span> <span class="caps">PHP</span> installed whether you want it or not. At some point …</p><figure class="image-block"><img src="https://duncanlock.net/images/posts/finally-figured-out-my-mysterious-418unused-http-code-dreamhost/teapot.png" alt="Blueprint style diagram showing a Teapot.">
<figcaption>Figure 1. A teapot, cut in half. Sort of. Original clipart Kitchen Utensils Silhouette, by <span class="caps">GDJ</span>, Public Domain. <a href="https://en.wikipedia.org/wiki/Teapot">More on Teapots</a>.</figcaption></figure>
<p>I’ve had a mysterious broken page on this site for a while - but been too busy to look into it. My <a href="https://duncanlock.net/blog/2013/08/27/comprehensive-linux-backups-with-etckeeper-backupninja/">Comprehensive Linux Backups with etckeeper <span class="amp">&</span> backupninja</a> article has been refusing to load, and returning a weird <span class="caps">HTTP</span> 418 Unused status code instead. I finally made the time to figure out the cause.</p>
<p>It turned out that this was being caused by the Apache/<span class="caps">PHP</span> <code>mod_security</code> module. This is a static website - there’s no <span class="caps">PHP</span> anywhere - so why would that be a problem? Well, so far I’ve been very happily hosting the site on my old DreamHost shared hosting account - which comes with Apache <span class="amp">&</span> <span class="caps">PHP</span> installed whether you want it or not. At some point I must have checked the <a href="https://help.dreamhost.com/hc/en-us/articles/215947927">‘Extra Web Security?’ option in the DreamHost control panel</a> for this domain.</p>
<section class="doc-section level-1"><h2 id="_checking_the_logs">Checking the Logs</h2><p>I <span class="caps">SSH</span>’d in to my account on the shared server to look at the logs. The apache <code>error.log</code> for the site had lots of the following entries:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="log">[Tue Mar 22 01:17:39 2016] [error] [client 24.84.4.121] ModSecurity: Access denied with code 418 (phase 4). Pattern match "(?:<title>[^<]*?(?:\\\\b(?:(?:c(?:ehennemden|gi-telnet)|gamma web shell)\\\\b|imhabi
rligi phpftp)|(?:r(?:emote explorer|57shell)|aventis klasvayv|zehir)\\\\b|\\\\.::(?:news remote php shell injection::\\\\.| rhtools\\\\b)|ph(?:p(?:(?: commander|-terminal)\\\\b|remot ..." at RESPONSE_BODY. [
file "/dh/apache2/template/etc/mod_sec2/modsecurity_crs_45_trojans.conf"] [line "34"] [id "950922"] [msg "Backdoor access"] [severity "CRITICAL"] [tag "MALICIOUS_SOFTWARE/TROJAN"] [hostname "duncanlock.net"]
[uri "/blog/2013/08/27/comprehensive-linux-backups-with-etckeeper-backupninja/index.html"] [unique_id "VvD-o9BxuqUAAGqPBDYAAAAA"]</code></pre></div>
<p>Line 34 of the <code>.conf</code> file that it references:</p>
<div class="listing-block"><pre class="rouge highlight"><code> /dh/apache2/template/etc/mod_sec2/modsecurity_crs_45_trojans.conf</code></pre></div>
<p>looks like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="apache">SecRule RESPONSE_BODY "(?:<title>[^<]*?(?:\b(?:(?:c(?:ehennemden|gi-telnet)|gamma web shell)\b|imhabirligi phpftp)|(?:r(?:emote explorer|57shell)|aventis klasvayv|zehir)\b|\.::(?:news remote php shell injection::\.| rhtools\b)|ph(?:p(?:(?: commander|-terminal)\b|remoteview)|vayv)|myshell)|\b(?:(?:(?:microsoft windows\b.{,10}?\bversion\b.{,20}?\(c\) copyright 1985-.{,10}?\bmicrosoft corp|ntdaddy v1\.9 - obzerve \| fux0r inc)\.|(?:www\.sanalteror\.org - indexer <span class="ss">and</span> read|haxplor)er|php(?:konsole| shell)|c99shell)\b|aventgrup\.<br>|drwxr))" \
<span class="err">"</span>phase:4,ctl:auditLogParts=+E,deny,log,auditlog,msg:'Backdoor access',id:'950922',tag:'MALICIOUS_SOFTWARE/TROJAN',severity:'2'"</code></pre></div>
<p>This rather large regular expression <em>just happened</em> to match a perfectly innocent piece of text in my <a href="https://duncanlock.net/blog/2013/08/27/comprehensive-linux-backups-with-etckeeper-backupninja/">Comprehensive Linux Backups with etckeeper <span class="amp">&</span> backupninja</a> article - specifically, this directory listing matched the <code>|drwxr))</code> bit at the end of that regex:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo ls</span> <span class="nt">-lah</span> /etc/backup.d/
<span class="go">
total 40K
drwxrwx--- 2 root root 4.0K May 19 16:54 .
drwxr-xr-x 154 root root 12K May 19 15:25 ..
-rw------- 1 root root 1.4K May 19 16:54 10-little-things.sh
</span><span class="c">...</span></code></pre></div>
<p>This caused Apache’s <a href="https://github.com/SpiderLabs/ModSecurity"><code>mod_security</code></a> to eat requests for that page, and DreamHost to return a 418 error instead:</p>
<div class="quote-block"><blockquote><p>This code was defined in 1998 as one of the traditional <a href="https://en.wikipedia.org/wiki/Internet_Engineering_Task_Force"><span class="caps">IETF</span></a> <a href="https://en.wikipedia.org/wiki/April_Fools%27_Day_RFC">April Fools’ jokes</a>, in <a href="https://tools.ietf.org/html/rfc2324"><span class="caps">RFC</span> 2324</a>, <a href="https://en.wikipedia.org/wiki/Hyper_Text_Coffee_Pot_Control_Protocol">Hyper Text Coffee Pot Control Protocol</a>, and is not expected to be implemented by actual <span class="caps">HTTP</span> servers. The <span class="caps">RFC</span> specifies this code should be returned by tea pots requested to brew coffee. This <span class="caps">HTTP</span> status is used as an easter egg in some websites, including Google.com.</p><footer>— <cite><a class="bare" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/418">https://developer.mozilla.org/en-<span class="caps">US</span>/docs/Web/<span class="caps">HTTP</span>/Status/418</a></cite></footer></blockquote></div>
<p>I guess it’s easy to see in logs and is unlikely to be caused by anything else. Took me a while to figure out where it was coming from, though.</p></section>
<section class="doc-section level-1"><h2 id="_fixing_it">Fixing it</h2><p>The simplest fix for this is to just uncheck the <a href="https://help.dreamhost.com/hc/en-us/articles/215947927">‘Extra Web Security?’ option in the DreamHost control panel</a> for this domain. You have to wait a few minutes for the change to happen, but after that, the page should start working.</p>
<p>What I <em>actually</em> decided to do was to take advantage of the static nature of the site - and experiment with hosting this blog directly on Amazon S3. I’ll cover that in another article, if I decide to stick with it.</p></section>How I upgraded this website to Pelican 3.62016-03-05T01:51:54-08:002016-03-05T01:51:54-08:00Duncan Locktag:duncanlock.net,2016-03-05:/blog/2016/03/05/how-i-upgraded-this-website-to-pelican-36/<p>This site has been generated using Pelican 3.3 for over two years - and I finally found some time to upgrade to the current version of Pelican, 3.6.3. This is how I did the upgrade.</p>
<p>I decided to be lazy and do the upgrade in-place, instead of creating a new <code>virtualenv</code> and copying the content <span class="amp">&</span> settings over. Luckily, this worked out <span class="caps">OK</span>, after a bit of fiddling around.</p>
<p>I also decided, rather cavalierly, to upgrade all the packages in the <code>virtualenv</code> to their latest versions while I was at it. To do this, I upgraded <code>pip</code>, then used <a href="https://pypi.python.org/pypi/pip-review">pip-review</a>. To upgrade <code>pip</code> <span class="amp">&</span> install <code>pip-review</code> system wide, run this on the command line:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo</span> <span class="nt">-H</span> pip <span class="nb">install</span> <span class="nt">--upgrade</span> pip
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo</span> <span class="nt">-H</span> pip <span class="nb">install </span>pip-review</code></pre></div>
<p>Then upgrade eveything in the sites <code>virtualenv</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>libjpeg-dev
<span class="gp">$</span><span class="w"> </span>workon duncanlock.net
<span class="gp">$</span><span class="w"> </span>pip-review …</code></pre></div><p>This site has been generated using Pelican 3.3 for over two years - and I finally found some time to upgrade to the current version of Pelican, 3.6.3. This is how I did the upgrade.</p>
<p>I decided to be lazy and do the upgrade in-place, instead of creating a new <code>virtualenv</code> and copying the content <span class="amp">&</span> settings over. Luckily, this worked out <span class="caps">OK</span>, after a bit of fiddling around.</p>
<p>I also decided, rather cavalierly, to upgrade all the packages in the <code>virtualenv</code> to their latest versions while I was at it. To do this, I upgraded <code>pip</code>, then used <a href="https://pypi.python.org/pypi/pip-review">pip-review</a>. To upgrade <code>pip</code> <span class="amp">&</span> install <code>pip-review</code> system wide, run this on the command line:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo</span> <span class="nt">-H</span> pip <span class="nb">install</span> <span class="nt">--upgrade</span> pip
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo</span> <span class="nt">-H</span> pip <span class="nb">install </span>pip-review</code></pre></div>
<p>Then upgrade eveything in the sites <code>virtualenv</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>libjpeg-dev
<span class="gp">$</span><span class="w"> </span>workon duncanlock.net
<span class="gp">$</span><span class="w"> </span>pip-review <span class="nt">--auto</span>
<span class="gp">$</span><span class="w"> </span>pip freeze <span class="o">></span> requirements.txt</code></pre></div>
<p>Installing <code><strong>libjpeg-dev</strong></code> was a requirement for upgrading Pillow, I think.</p>
<section class="doc-section level-1"><h2 id="_upgrade_settings">Upgrade Settings</h2><p>I then tried to generate the site – and got this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pelican content
<span class="go">
WARNING: PLUGIN_PATH setting has been replaced by PLUGIN_PATHS, moving it to the new setting name.
WARNING: Defining PLUGIN_PATHS setting as string has been deprecated (should be a list)
WARNING: Deprecated setting ARTICLE_DIR, moving it to ARTICLE_PATHS list
WARNING: Deprecated setting PAGE_DIR, moving it to PAGE_PATHS list</span></code></pre></div>
<p>These are all warnings about deprecated settings in my ancient <code>pelicanconf.py</code> settings file. I fixed those by making these changes:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="diff"><span class="gd">-ARTICLE_DIR = ('posts')
-PAGE_DIR = ('pages')
</span><span class="gi">+ARTICLE_PATHS = ['posts']
+PAGE_PATHS = ['pages']
</span><span class="err">
</span><span class="gd">-PLUGIN_PATH = '../pelican-plugins'
</span><span class="gi">+PLUGIN_PATHS = ['../pelican-plugins']</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_upgrade_plugins">Upgrade Plugins</h2><p>I’m sourcing my plugins from a git checkout of the <a href="https://github.com/getpelican/pelican-plugins">pelican-plugins repository</a>, so I pulled that up to date like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">cd</span> ../pelican-plugins/
<span class="gp">$</span><span class="w"> </span>git fetch origin
<span class="gp">$</span><span class="w"> </span>git co master
<span class="gp">$</span><span class="w"> </span>git pull <span class="nt">--recurse-submodules</span> <span class="o">&&</span> git submodule update <span class="nt">--recursive</span></code></pre></div>
<p>The only change precipitated by this was a minor tweak to use the <code>series</code> plugin instead of the deprecated <code>multipart</code> one. I made this change to the theme:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="diff"><span class="gd">--- a/templates/article-sidebar-multipart.html
</span><span class="gi">+++ b/templates/article-sidebar-multipart.html
</span><span class="p">@@ -1,9 +1,9 @@</span>
<span class="gd">-{% if article.metadata.parts_articles %}
</span><span class="gi">+{% if article.series %}
</span> <div class="row-fluid">
<nav>
<p>This post is part of a series:</p>
<ol class="parts">
<span class="gd">- {% for part_article in article.metadata.parts_articles %}
</span><span class="gi">+ {% for part_article in article.series.all %}
</span> <li {% if part_article == article %}class="active"{% endif %}>
{% if part_article == article %}
{{ part_article.title }}
<span class="p">@@ -15,4 +15,4 @@</span>
</ol>
</nav>
</div>
{% endif %}</code></pre></div>
<p>and changed my config to load the <code>series</code> plugin instead:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="c1"># Which plugins to enable
</span><span class="n">PLUGINS</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">'better_figures_and_images'</span><span class="p">,</span>
<span class="s">'assets'</span><span class="p">,</span>
<span class="s">'related_posts'</span><span class="p">,</span>
<span class="s">'extract_toc'</span><span class="p">,</span>
<span class="s">'post_stats'</span><span class="p">,</span>
<span class="s">'series'</span>
<span class="p">]</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_minor_tweak_to_syntax_highlighting_in_blueprint_theme">Minor tweak to syntax highlighting in blueprint theme</h2><p>As my pygments module had got a <a href="http://pygments.org/docs/changelog/">major version bump from 1.6 to 2.1.2</a>, I updated the pygments <span class="caps">CSS</span> files included with the theme. To do this, I ran this at the command line, in the website folder, then merged the result into the existing <code>pygments-monokai.css</code> file in the blueprint themes <code>static/css</code> folder:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pygmentize <span class="nt">-S</span> monokai <span class="nt">-f</span> html <span class="nt">-a</span> .highlight | <span class="nb">sort</span> <span class="o">></span> pygments-monokai.css</code></pre></div>
<p>I also had an existing <code>pygments.css</code> in there for some reason, which had a few extra styles in. I merged these into <code>pygments-monokai.css</code> and deleted it, so I could load just <span class="caps">CSS</span> one file.</p></section>
<section class="doc-section level-1"><h2 id="_new_feature_caching">New feature: Caching</h2><p>Pelican 3.6 now has build caching, which 3.3 didn’t. To take advantage of this, I set these properties in my settings file:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="c1">#################################
#
# Cache Settings
#
#################################
</span>
<span class="n">CACHE_CONTENT</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">CHECK_MODIFIED_METHOD</span> <span class="o">=</span> <span class="s">'mtime'</span>
<span class="n">LOAD_CONTENT_CACHE</span> <span class="o">=</span> <span class="bp">True</span>
<span class="n">GZIP_CACHE</span> <span class="o">=</span> <span class="bp">False</span></code></pre></div>
<p>Doing this cut the generation time for this site roughly in half – from ~13 seconds, down to ~7 seconds - a worthwhile improvement. Symlinking the <code>./cache</code> folder to my <span class="caps">SSD</span> instead of the regular <span class="caps">HD</span>…​ didn’t make much difference to the time. Symlinking it to a folder on <a href="https://wiki.archlinux.org/index.php/Tmpfs">a tmpfs <span class="caps">RAM</span> disk</a> didn’t seem to make much difference either – so for this little site, the caching doesn’t seem very <span class="caps">IO</span> bound, which was a little unexpected. Maybe this is because the source files are still on a regular <span class="caps">HD</span> - or maybe the <span class="caps">OS</span> was already doing a good job with disk caching.</p></section>Reliably Building VirtualBox Guest Additions on CentOS 6.x2014-01-22T19:54:48-08:002021-06-11T11:12:15-07:00Duncan Locktag:duncanlock.net,2014-01-22:/blog/2014/01/22/reliably-building-virtualbox-guest-additions-on-centos-6x/<p>We use CentOS VMs at work to emulate our production environment - and it took me a while to figure out how to get the VirtualBox Guest Additions to build reliably on CentOS 6.4/5. This is what I’ve currently settled on as a reliable method.</p>
<p>First, make sure that you’ve got the kernel headers and tools installed that you need to build stuff:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>yum update <span class="nt">-y</span>
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>yum <span class="nb">install </span>gcc kernel-devel kernel-headers dkms make bzip2 perl</code></pre></div>
<p>Make sure that you’ve only got the current set of kernel headers installed - the one for the kernel you’re actually running. Having more than one set installed prevents this working properly. Running this should show you one version of each kernel package:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>rpm <span class="nt">-qa</span> | <span class="nb">grep </span>kernel | <span class="nb">sort</span></code></pre></div>
<p>It should look something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">dracut-kernel-004-336.el6_5.2.noarch …</code></pre></div><p>We use CentOS VMs at work to emulate our production environment - and it took me a while to figure out how to get the VirtualBox Guest Additions to build reliably on CentOS 6.4/5. This is what I’ve currently settled on as a reliable method.</p>
<p>First, make sure that you’ve got the kernel headers and tools installed that you need to build stuff:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>yum update <span class="nt">-y</span>
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>yum <span class="nb">install </span>gcc kernel-devel kernel-headers dkms make bzip2 perl</code></pre></div>
<p>Make sure that you’ve only got the current set of kernel headers installed - the one for the kernel you’re actually running. Having more than one set installed prevents this working properly. Running this should show you one version of each kernel package:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>rpm <span class="nt">-qa</span> | <span class="nb">grep </span>kernel | <span class="nb">sort</span></code></pre></div>
<p>It should look something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">dracut-kernel-004-336.el6_5.2.noarch
kernel-2.6.32-431.3.1.el6.x86_64
kernel-devel-2.6.32-431.3.1.el6.x86_64
kernel-firmware-2.6.32-431.3.1.el6.noarch
kernel-headers-2.6.32-431.3.1.el6.x86_64</code></pre></div>
<p>if you have multiple versions of one of the kernel packages installed, it will look something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">dracut-kernel-004-336.el6_5.2.noarch
kernel-2.6.32-132.1.2.el6.x86_64
kernel-2.6.32-431.3.1.el6.x86_64
kernel-devel-2.6.32-132.1.2.el6.x86_64
kernel-devel-2.6.32-431.3.1.el6.x86_64
kernel-firmware-2.6.32-431.3.1.el6.noarch
kernel-headers-2.6.32-431.3.1.el6.x86_64</code></pre></div>
<p>remove the older ones:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>rpm <span class="nt">-e</span> kernel-2.6.32-132.1.2.el6.x86_64 kernel-devel-2.6.32-132.1.2.el6.x86_64</code></pre></div>
<p>Now we’re ready to install <span class="amp">&</span> build the guest additions, so mount the <span class="caps">CD</span> by selecting ‘Install Guest Additions’ from the VirtualBox menu. Don’t autorun it - you’ll need to run the following to install it properly.</p>
<p>Some of this needs to run as sudo, and we need to export variables that sudo can see, so we’ll just do all of the following as root:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">sudo </span>su -
<span class="nb">export </span><span class="nv">MAKE</span><span class="o">=</span><span class="s1">'/usr/bin/gmake -i'</span>
<span class="nb">export </span><span class="nv">KERN_DIR</span><span class="o">=</span>/usr/src/kernels/<span class="sb">`</span><span class="nb">uname</span> <span class="nt">-r</span><span class="sb">`</span>
<span class="c"># Change the version number here to whatever is current</span>
<span class="c"># cd /media/VB<tab> will probably work</span>
<span class="nb">cd</span> /media/VBOXADDITIONS_4.3.6_91406/
./VBoxLinuxAdditions.run</code></pre></div>
<p>This should build <span class="amp">&</span> install all the guest additions successfully. If it doesn’t, let me know in the comments.</p>It was a Good Week2013-11-22T02:57:06-08:002021-06-10T21:08:52-07:00Duncan Locktag:duncanlock.net,2013-11-22:/blog/2013/11/22/it-was-a-good-week/<figure class="image-block"><img src="https://duncanlock.net/images/posts/it-was-a-good-week/IMG_20131119_095456-small.jpg" alt="IMG 20131119 095456 small">
<figcaption>Figure 1. Taken from my Wife’s office window, around 10am.</figcaption></figure>
<p>The weather was great this week - crisp and autumnal, with beautiful sunshine all week. It snowed a little on the local mountains and <a href="http://cypressmountain.com/">Cypress Mountain</a> has opened for (limited) Skiing already, using the lower temperatures to make lots of extra snow. Whistler also opened this week, along with Grouse.</p>
<figure class="image-block"><a class="image" href="https://en.wikipedia.org/wiki/Golden_hour_(photography)"><img src="https://duncanlock.net/images/posts/it-was-a-good-week/IMG_20131119_161127-small.jpg" alt="IMG 20131119 161127 small"></a>
<figcaption>Figure 2. Taken from my Wife’s office window in the afternoon, during the Golden Hour, which is now around 4pm.</figcaption></figure>
<p>In other, even better news, I’m starting a fantastic new job with <a href="http://www.phemi.com/about-us/"><span class="caps">PHEMI</span> Health Systems</a> on Monday - allowing me to afford to actually go skiing. I’ll be working as a front-end web developer, helping to build their healthcare information system, directly helping to improve clinical productivity, patient outcomes and medical research. Very excited to be using my …</p><figure class="image-block"><img src="https://duncanlock.net/images/posts/it-was-a-good-week/IMG_20131119_095456-small.jpg" alt="IMG 20131119 095456 small">
<figcaption>Figure 1. Taken from my Wife’s office window, around 10am.</figcaption></figure>
<p>The weather was great this week - crisp and autumnal, with beautiful sunshine all week. It snowed a little on the local mountains and <a href="http://cypressmountain.com/">Cypress Mountain</a> has opened for (limited) Skiing already, using the lower temperatures to make lots of extra snow. Whistler also opened this week, along with Grouse.</p>
<figure class="image-block"><a class="image" href="https://en.wikipedia.org/wiki/Golden_hour_(photography)"><img src="https://duncanlock.net/images/posts/it-was-a-good-week/IMG_20131119_161127-small.jpg" alt="IMG 20131119 161127 small"></a>
<figcaption>Figure 2. Taken from my Wife’s office window in the afternoon, during the Golden Hour, which is now around 4pm.</figcaption></figure>
<p>In other, even better news, I’m starting a fantastic new job with <a href="http://www.phemi.com/about-us/"><span class="caps">PHEMI</span> Health Systems</a> on Monday - allowing me to afford to actually go skiing. I’ll be working as a front-end web developer, helping to build their healthcare information system, directly helping to improve clinical productivity, patient outcomes and medical research. Very excited to be using my skills for something worthwhile <span class="amp">&</span> interesting - while working with an amazing group of very smart people.</p>
<p>I am a very, very luck guy - and I have a lot to be thankful for.</p>How to create thumbnails for PDFs with ImageMagick on Linux2013-11-18T20:05:38-08:002021-06-11T11:16:15-07:00Duncan Locktag:duncanlock.net,2013-11-18:/blog/2013/11/18/how-to-create-thumbnails-for-pdfs-with-imagemagick-on-linux/<p>To create image thumbnails from a <span class="caps">PDF</span> document, run this in a terminal window:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>convert <span class="nt">-thumbnail</span> x300 <span class="nt">-background</span> white <span class="nt">-alpha</span> remove input_file.pdf[0] output_thumbnail.png</code></pre></div>
<p>The parameters to <a href="http://www.imagemagick.org/script/command-line-options.php">convert</a> do the following things:</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 25%;"><col style="width: 75%;"></colgroup><thead><tr><th class="halign-left valign-top">Parameter</th><th class="halign-left valign-top">Effect</th></tr></thead><tbody><tr><td class="halign-left valign-top"><code>-thumbnail</code></td><td class="halign-left valign-top">Similar to -resize, but optimized for speed and strips metadata.</td></tr><tr><td class="halign-left valign-top"><code>x300</code></td><td class="halign-left valign-top">Make the thumbnail 300px tall, and whatever width maintains the aspect ratio.</td></tr><tr><td class="halign-left valign-top"><code>-background white</code></td><td class="halign-left valign-top">Sets the thumbnail background to white.</td></tr><tr><td class="halign-left valign-top"><code>-alpha remove</code></td><td class="halign-left valign-top">Removes the alpha channel from the thumbnail output.</td></tr><tr><td class="halign-left valign-top"><code>input_file.pdf</code></td><td class="halign-left valign-top">The <span class="caps">PDF</span> file to use as input.</td></tr><tr><td class="halign-left valign-top"><code>[0]</code></td><td class="halign-left valign-top">The page number of the input file to use for the thumbnail.</td></tr><tr><td class="halign-left valign-top"><code>output_thumbnail.png</code></td><td class="halign-left valign-top">The output thumbnail file to create.</td></tr></tbody></table></div>
<p>If you want larger thumbnails, just change the <code>x300</code> parameter to match. If you want to output .jpg’s (or anything else, like .gif), just change the file …</p><p>To create image thumbnails from a <span class="caps">PDF</span> document, run this in a terminal window:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>convert <span class="nt">-thumbnail</span> x300 <span class="nt">-background</span> white <span class="nt">-alpha</span> remove input_file.pdf[0] output_thumbnail.png</code></pre></div>
<p>The parameters to <a href="http://www.imagemagick.org/script/command-line-options.php">convert</a> do the following things:</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 25%;"><col style="width: 75%;"></colgroup><thead><tr><th class="halign-left valign-top">Parameter</th><th class="halign-left valign-top">Effect</th></tr></thead><tbody><tr><td class="halign-left valign-top"><code>-thumbnail</code></td><td class="halign-left valign-top">Similar to -resize, but optimized for speed and strips metadata.</td></tr><tr><td class="halign-left valign-top"><code>x300</code></td><td class="halign-left valign-top">Make the thumbnail 300px tall, and whatever width maintains the aspect ratio.</td></tr><tr><td class="halign-left valign-top"><code>-background white</code></td><td class="halign-left valign-top">Sets the thumbnail background to white.</td></tr><tr><td class="halign-left valign-top"><code>-alpha remove</code></td><td class="halign-left valign-top">Removes the alpha channel from the thumbnail output.</td></tr><tr><td class="halign-left valign-top"><code>input_file.pdf</code></td><td class="halign-left valign-top">The <span class="caps">PDF</span> file to use as input.</td></tr><tr><td class="halign-left valign-top"><code>[0]</code></td><td class="halign-left valign-top">The page number of the input file to use for the thumbnail.</td></tr><tr><td class="halign-left valign-top"><code>output_thumbnail.png</code></td><td class="halign-left valign-top">The output thumbnail file to create.</td></tr></tbody></table></div>
<p>If you want larger thumbnails, just change the <code>x300</code> parameter to match. If you want to output .jpg’s (or anything else, like .gif), just change the file extension on the <code>output_thumbnail.png</code> parameter. If you leave the <code>[0]</code> off the end of the input filename, you’ll get a thumbnail for each page, not just the first; setting it to <code>[1]</code> will get you a thumbnail of the second page, and so on…​</p>
<section class="doc-section level-1"><h2 id="_do_a_whole_folder_at_a_time">Do a whole folder at a time</h2><p>To create thumbnails for a whole folder of <span class="caps">PDF</span>’s, do this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="k">for </span>f <span class="k">in</span> <span class="k">*</span>.pdf<span class="p">;</span> <span class="k">do </span>convert <span class="nt">-thumbnail</span> x300 <span class="nt">-background</span> white <span class="nt">-alpha</span> remove <span class="s2">"</span><span class="nv">$f</span><span class="s2">"</span><span class="o">[</span>0] <span class="s2">"</span><span class="k">${</span><span class="nv">f</span><span class="p">%.pdf</span><span class="k">}</span><span class="s2">.png"</span><span class="p">;</span> <span class="k">done</span></code></pre></div>
<p>This will create thumbnails with filenames that match the input <span class="caps">PDF</span>’s, except with an extra <code>.pdf</code> extension.</p></section>
<section class="doc-section level-1"><h2 id="_requirements">Requirements</h2><p>This requires ImageMagick and Ghostscript, which you can install like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>imagemagick ghostscript</code></pre></div>
<p>This technique should also work on a Mac, provided you have these both installed.</p></section>The Better Figures & Images Pelican plugin now supports Figure Numbering2013-10-19T22:08:26-07:002021-06-12T21:56:28-07:00Duncan Locktag:duncanlock.net,2013-10-19:/blog/2013/10/19/the-better-figures-images-pelican-plugin-now-supports-figure-numbering/<figure class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-200x200.png" alt="A dummy placeholder image, 200x200 pixels square.">
<figcaption>Figure 1. This figure has automatic figure numbering.</figcaption></figure>
<p>I had a feature request for automatic figure numbering, like latex. I was revamping this plugin for Pelican 3.3 anyway - and this didn’t seem too hard - so I decided to add it.</p>
<p>So, the <a href="https://duncanlock.net/blog/2013/05/29/better-figures-images-plugin-for-pelican/">Better Figures <span class="amp">&</span> Images plugin</a> now supports automatic figure numbering. To enable this for all posts, just add this to your config file:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">FIGURE_NUMBERS</span> <span class="o">=</span> <span class="bp">True</span></code></pre></div>
<p>If you want to enable this per post, just add this to the metadata at the top of the post:</p>
<p>for restructuredText add this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">:figure_numbers: true</code></pre></div>
<p>and for Markdown add this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="markdown">figure_numbers: true</code></pre></div>
<section class="admonition-block caution" role="doc-notice"><h6 class="block-title label-only"><span class="title-label">Caution: </span></h6><p>Can you have Figures in Markdown?</p>
<p>I use reStructuredText for this site, and I’m not sure if you can even <em>have</em> Figures in Markdown documents - and I haven’t tested it, so caveat emptor …</p></section><figure class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-200x200.png" alt="A dummy placeholder image, 200x200 pixels square.">
<figcaption>Figure 1. This figure has automatic figure numbering.</figcaption></figure>
<p>I had a feature request for automatic figure numbering, like latex. I was revamping this plugin for Pelican 3.3 anyway - and this didn’t seem too hard - so I decided to add it.</p>
<p>So, the <a href="https://duncanlock.net/blog/2013/05/29/better-figures-images-plugin-for-pelican/">Better Figures <span class="amp">&</span> Images plugin</a> now supports automatic figure numbering. To enable this for all posts, just add this to your config file:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">FIGURE_NUMBERS</span> <span class="o">=</span> <span class="bp">True</span></code></pre></div>
<p>If you want to enable this per post, just add this to the metadata at the top of the post:</p>
<p>for restructuredText add this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">:figure_numbers: true</code></pre></div>
<p>and for Markdown add this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="markdown">figure_numbers: true</code></pre></div>
<section class="admonition-block caution" role="doc-notice"><h6 class="block-title label-only"><span class="title-label">Caution: </span></h6><p>Can you have Figures in Markdown?</p>
<p>I use reStructuredText for this site, and I’m not sure if you can even <em>have</em> Figures in Markdown documents - and I haven’t tested it, so caveat emptor.</p></section>
<p>It adds this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><span</span> <span class="na">class=</span><span class="s">"fig_num"</span> <span class="na">id=</span><span class="s">"fig_X"</span><span class="nt">></span>Figure X: <span class="nt"></span></span></code></pre></div>
<p>to the start of the figure captions, where <code>X</code> is the current figure number. It only does this to figures that already have captions - it’ll skip figures without. It completely ignores images, even it they have captions - it only affects figures.</p>
<p>The markup it outputs looks something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><div</span> <span class="na">class=</span><span class="s">"figure"</span> <span class="na">style=</span><span class="s">"width: 250px; max-width: 100%; height: auto;"</span><span class="nt">></span>
<span class="nt"><img</span> <span class="na">style=</span><span class="s">"width: 250px; max-width: 100%; height: auto;"</span> <span class="na">alt=</span><span class="s">""</span> <span class="na">src=</span><span class="s">"/images/image.jpg"</span> <span class="nt">/></span>
<span class="nt"><p</span> <span class="na">class=</span><span class="s">"caption"</span><span class="nt">></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"fig_num"</span> <span class="na">id=</span><span class="s">"fig_1"</span><span class="nt">></span>Figure 1: <span class="nt"></span></span>This is the caption of the figure.
<span class="nt"></p></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"legend"</span><span class="nt">></span>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.
<span class="nt"></div></span>
<span class="nt"></div></span></code></pre></div>
<p>The figure number on it’s own looks something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><span</span> <span class="na">class=</span><span class="s">"fig_num"</span> <span class="na">id=</span><span class="s">"fig_1"</span><span class="nt">></span>Figure 1: <span class="nt"></span></span></code></pre></div>
<p>This allows you to style it in <span class="caps">CSS</span> using the class and to link to it using the id/<code>#fig_1</code> anchor.</p>
<aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>Automatic Figure numbering is new and isn’t upstream yet - check out the <code>figure_numbers</code> branch from my git repo, <a href="https://github.com/dflock/pelican-plugins/tree/figure_numbers">here</a> if you want to use it.</p>
<p>While we’re on that subject, this plugin <a href="https://duncanlock.net/blog/2013/10/18/how-i-upgraded-this-website-to-pelican-33/">does work with Pelican 3.3</a>, but that’s not upstream yet either - the <code>figure_numbers</code> branch includes those fixes too.</p>
<p>I’m trying to get this upstream soonest - and will update here when done.</p></aside>
<section class="doc-section level-1"><h2 id="_the_results_look_like_this">The results look like this</h2><p>Here are a few working examples, showing the results of using the plugin. The original rst source for these are available in the plugins <code>/test</code> folder:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-800x300.png" alt="dummy 800x300">
<figcaption>Figure 2. This image is wider than the column it’s in - try resizing the browser window. Because of the max-width: 100%, the image is resized to fit the column.</figcaption></figure>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-200x200.png" alt="A dummy placeholder image" width="200x200 pixels square.">
<figcaption>Figure 3. This image is only 200px wide - smaller that the column it’s in. The max-width: 100% doesn’t stretch the image, because it’s also got a width: 200px - making it shrink to fit.</figcaption></figure>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur.</p>
<section class="paragraph"><h6 class="block-title">This is the third image caption. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod</h6><p class="align-right">tempor incididunt ut labore et dolore magna aliqua.
image::{static}/images/posts/better-figures-images-plugin-for-pelican/dummy-250x300.png[map to buried treasure 2]</p></section>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-200x200.png" alt="dummy 200x200"></div>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<section class="paragraph"><h6 class="block-title">This is the fourth image caption. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod</h6><p class="align-right">tempor incididunt ut labore et dolore magna aliqua.
image::{static}/images/posts/better-figures-images-plugin-for-pelican/dummy-250x300.png[map to buried treasure 3]</p></section>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p></section>How I upgraded this website to Pelican 3.32013-10-18T17:08:14-07:002021-06-12T09:58:09-07:00Duncan Locktag:duncanlock.net,2013-10-18:/blog/2013/10/18/how-i-upgraded-this-website-to-pelican-33/<p>There are <a href="https://github.com/getpelican/pelican/issues?milestone=5&state=closed">quite a few changes</a> in <a href="http://blog.getpelican.com/pelican-3.3-released.html">Pelican 3.3</a> - most of them minor, but a few which might mean making some changes to your site in order to upgrade. This is what I did to move my site from Pelican 3.2 to 3.3.</p>
<p>The change that had the biggest impact and took the most work was around image linking - caused by a combination of things. I think I was doing it wrong before and things changed in a way that meant this no longer worked. I also had to update my <a href="https://duncanlock.net/blog/2013/05/29/better-figures-images-plugin-for-pelican/">Better Figures <span class="amp">&</span> Images plugin</a> to take this into account.</p>
<p>Previously, I’d been linking to my images like this, both in my theme:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Theme CSS Image Link */</span>
<span class="nt">background</span><span class="o">:</span> <span class="err">#2</span><span class="nt">C71B8</span> <span class="nt">url</span><span class="o">(/</span><span class="nt">static</span><span class="o">/</span><span class="nt">images</span><span class="o">/</span><span class="nt">blueprint-background</span><span class="nc">.png</span><span class="o">)</span> <span class="nt">repeat</span><span class="o">;</span></code></pre></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="c">{# Theme HTML/Jinja image link #}</span>
<span class="nt"><meta</span> <span class="na">name=</span><span class="s">"twitter:image"</span> <span class="na">content …</span></code></pre></div><p>There are <a href="https://github.com/getpelican/pelican/issues?milestone=5&state=closed">quite a few changes</a> in <a href="http://blog.getpelican.com/pelican-3.3-released.html">Pelican 3.3</a> - most of them minor, but a few which might mean making some changes to your site in order to upgrade. This is what I did to move my site from Pelican 3.2 to 3.3.</p>
<p>The change that had the biggest impact and took the most work was around image linking - caused by a combination of things. I think I was doing it wrong before and things changed in a way that meant this no longer worked. I also had to update my <a href="https://duncanlock.net/blog/2013/05/29/better-figures-images-plugin-for-pelican/">Better Figures <span class="amp">&</span> Images plugin</a> to take this into account.</p>
<p>Previously, I’d been linking to my images like this, both in my theme:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Theme CSS Image Link */</span>
<span class="nt">background</span><span class="o">:</span> <span class="err">#2</span><span class="nt">C71B8</span> <span class="nt">url</span><span class="o">(/</span><span class="nt">static</span><span class="o">/</span><span class="nt">images</span><span class="o">/</span><span class="nt">blueprint-background</span><span class="nc">.png</span><span class="o">)</span> <span class="nt">repeat</span><span class="o">;</span></code></pre></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="c">{# Theme HTML/Jinja image link #}</span>
<span class="nt"><meta</span> <span class="na">name=</span><span class="s">"twitter:image"</span> <span class="na">content=</span><span class="s">"</span><span class="cp">{{</span> <span class="nv">SITEURL</span> <span class="cp">}}</span><span class="s">/static/images/favicon-128x128.png"</span><span class="nt">></span></code></pre></div>
<p>and in my content:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">.. image:: /static/images/dunc_smiling_192x192.jpg
.. figure:: /static/images/pages/404-error.png</code></pre></div>
<p>This is linking to the output location of the final images, in the <code>/output/static/images</code> folder. I don’t think this is the way you’re supposed to do it - and it doesn’t appear to work with Pelican 3.3.</p>
<p>The way you’re supposed to do it, I think, is to use a place-holder for the first bit of the path and, effectively, link to the source location of the file, not the destination. The <a href="http://docs.getpelican.com/en/3.3.0/getting_started.html#linking-to-internal-content">relevant documentation is here</a> - and the above examples now look like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Theme CSS Image Link - not processed, so no place-holder */</span>
<span class="nt">background</span><span class="o">:</span> <span class="err">#2</span><span class="nt">C71B8</span> <span class="nt">url</span><span class="o">(/</span><span class="nt">images</span><span class="o">/</span><span class="nt">blueprint-background</span><span class="nc">.png</span><span class="o">)</span> <span class="nt">repeat</span><span class="o">;</span></code></pre></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="jinja"><span class="c">{# Theme HTML/Jinja image link - not processed, so no place-holder #}</span>
<span class="nt"><meta</span> <span class="na">name=</span><span class="s">"twitter:image"</span> <span class="na">content=</span><span class="s">"</span><span class="cp">{{</span> <span class="nv">SITEURL</span> <span class="cp">}}</span><span class="s">/images/favicon-128x128.png"</span><span class="nt">></span></code></pre></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">.. image:: {static}/images/dunc_smiling_192x192.jpg
.. figure:: {static}/images/pages/404-error.png</code></pre></div>
<section class="doc-section level-1"><h2 id="_theme_changes">Theme Changes</h2><p>The theme changes were the same: I removed the <code>/static</code> from the start of any image links. Because the theme files aren’t content, they don’t get processed by Pelican, so you don’t use the <code>{static}</code> place-holder here. All the changes looked something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="diff"><span class="gd">- <meta name="twitter:image" content="{{ SITEURL }}/static/images/test.png">
</span><span class="gi">+ <meta name="twitter:image" content="{{ SITEURL }}/images/test.png"></span></code></pre></div>
<p>You can see the Git commit with all the theme changes <a href="https://github.com/dflock/blueprint/commit/bae678828b4535fcece8327c0f2dbae63bf4c92f">here</a>, <a href="https://github.com/dflock/blueprint">in the Blueprint Theme Repository</a>.</p></section>
<section class="doc-section level-1"><h2 id="_content_changes">Content Changes</h2><div class="quote-block"><blockquote><p>The syntax for linking to source content has been changed in order to ensure compatibility with Markdown and reST extensions. For example, the new syntax for Markdown: <code>[a link relative to content root]({static}/article1.md)</code>
The previous <code>|filename|/article1.md</code> syntax will continue to be supported for backwards compatibility. – <a href="http://blog.getpelican.com/pelican-3.3-released.html">Pelican Blog</a></p></blockquote></div>
<p>Again, this is the related to image linking - and is the same change as above. This:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">.. image:: /static/images/dunc_smiling_192x192.jpg
.. figure:: /static/images/pages/404-error.png
:target: /static/images/pages/404-error.png</code></pre></div>
<p>becomes this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">.. image:: {static}/images/dunc_smiling_192x192.jpg
.. figure:: {static}/images/pages/404-error.png
:target: {static}/images/pages/404-error.png</code></pre></div>
<p>You need to do this in every post that has images. Fortunately this was simple to search <span class="amp">&</span> replace. On some posts I also have an extra piece of metadata called <code>thumbnail</code>, that also needed updating. This isn’t processed by Pelican, so no place-holder here:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="diff"><span class="gd">-:thumbnail: /static/images/posts/post-name/image.jpg
</span><span class="gi">+:thumbnail: /images/posts/post-name/image.jpg</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_configuration_changes">Configuration Changes</h2><p>Since the <code>FILES_TO_COPY</code> setting has been deprecated, you should replace it with the <code>STATIC_PATHS</code> and <code>EXTRA_PATH_METADATA</code> <a href="http://docs.getpelican.com/en/3.3.0/settings.html#basic-settings">settings</a>. The relevant part of my settings file changed like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="diff"> # static paths will be copied under the same name
<span class="gd">-STATIC_PATHS = ["images"]
</span><span class="gi">+STATIC_PATHS = [
+ 'images',
+ 'extras'
+]
</span><span class="err">
</span> # A list of extra files to copy from the source to the destination
<span class="gd">-FILES_TO_COPY = (
- ('extras/.htaccess', '.htaccess'),
- ('extras/robots.txt', 'robots.txt'),
- ('extras/favicon.ico', 'favicon.ico'),
-)
</span><span class="gi">+EXTRA_PATH_METADATA = {
+ 'extras/.htaccess': {'path': '.htaccess'},
+ 'extras/robots.txt': {'path': 'robots.txt'},
+ 'extras/favicon.ico': {'path': 'favicon.ico'},
+}</span></code></pre></div>
<p>You can see the Git commit with all the <a href="https://github.com/dflock/duncanlock.net/commit/bcee8b830d45daad00ea9428a339459689a27cf5">content <span class="amp">&</span> configuration changes here</a>, in the <a href="https://github.com/dflock/duncanlock.net">site repository</a>.</p></section>
<section class="doc-section level-1"><h2 id="_plugin_changes">Plugin Changes</h2><p>A special case for me is the {static}/posts/tech/better-figures-and-images-plugin-for-pelican.adoc[Better Figures <span class="amp">&</span> Images plugin]. I use this plugin and I also wrote it - and it stopped working.</p>
<p>In order to debug it, I first added in some logging support. I added this at the top with the other imports:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="p">.</span><span class="n">getLogger</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span></code></pre></div>
<p>and then some of this further down to output the paths that the plugin was seeing:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">logger</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Better Fig. PATH: %s'</span><span class="p">,</span> <span class="n">instance</span><span class="p">.</span><span class="n">settings</span><span class="p">[</span><span class="s">'PATH'</span><span class="p">])</span>
<span class="n">logger</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Better Fig. img.src: %s'</span><span class="p">,</span> <span class="n">img</span><span class="p">[</span><span class="s">'src'</span><span class="p">])</span></code></pre></div>
<p>This made it easier to figure out what I needed to change. The path handling code in the plugin was never very good, so I changed it from this mess:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="c1"># TODO: Pretty sure this isn't the right way to do this, too hard coded.
# There must be a setting that I should be using?
</span><span class="n">src</span> <span class="o">=</span> <span class="n">instance</span><span class="p">.</span><span class="n">settings</span><span class="p">[</span><span class="s">'PATH'</span><span class="p">]</span> <span class="o">+</span> <span class="s">'/images/'</span> <span class="o">+</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="n">img</span><span class="p">[</span><span class="s">'src'</span><span class="p">])[</span><span class="mi">1</span><span class="p">]</span>
<span class="c1">#src = instance.settings['PATH'] + '/images/' + os.path.split(img['src'])[1]
</span>
<span class="c1"># The method mentioned above is only working if the images are really in the "images" folder.
# It's also not working on subdirectories inside the image folder
# Both issues are fixed:
# Changed the static "/images/" string to the proper path which is extracted from the 'split' tuple
# The first 7 letters are cutted ("/static") to get a valid link.
# Somehow the static folder isn't created in the output folder. It's only on the server after 'make ftp_upload'
</span><span class="n">src</span> <span class="o">=</span> <span class="n">instance</span><span class="p">.</span><span class="n">settings</span><span class="p">[</span><span class="s">'PATH'</span><span class="p">]</span> <span class="o">+</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="n">img</span><span class="p">[</span><span class="s">'src'</span><span class="p">])[</span><span class="mi">0</span><span class="p">][</span><span class="mi">7</span><span class="p">:]</span> <span class="o">+</span> <span class="s">'/'</span> <span class="o">+</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="n">img</span><span class="p">[</span><span class="s">'src'</span><span class="p">])[</span><span class="mi">1</span><span class="p">]</span></code></pre></div>
<p>to this slightly more robust mess:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">logger</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Better Fig. PATH: %s'</span><span class="p">,</span> <span class="n">instance</span><span class="p">.</span><span class="n">settings</span><span class="p">[</span><span class="s">'PATH'</span><span class="p">])</span>
<span class="n">logger</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Better Fig. img.src: %s'</span><span class="p">,</span> <span class="n">img</span><span class="p">[</span><span class="s">'src'</span><span class="p">])</span>
<span class="n">img_path</span><span class="p">,</span> <span class="n">img_filename</span> <span class="o">=</span> <span class="n">path</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="n">img</span><span class="p">[</span><span class="s">'src'</span><span class="p">])</span>
<span class="n">logger</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Better Fig. img_path: %s'</span><span class="p">,</span> <span class="n">img_path</span><span class="p">)</span>
<span class="n">logger</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Better Fig. img_fname: %s'</span><span class="p">,</span> <span class="n">img_filename</span><span class="p">)</span>
<span class="c1"># Strip off {static}, |filename| or /static
</span><span class="k">if</span> <span class="n">img_path</span><span class="p">.</span><span class="n">startswith</span><span class="p">((</span><span class="s">'{static}'</span><span class="p">,</span> <span class="s">'|filename|'</span><span class="p">)):</span>
<span class="n">img_path</span> <span class="o">=</span> <span class="n">img_path</span><span class="p">[</span><span class="mi">10</span><span class="p">:]</span>
<span class="k">elif</span> <span class="n">img_path</span><span class="p">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'/static'</span><span class="p">):</span>
<span class="n">img_path</span> <span class="o">=</span> <span class="n">img_path</span><span class="p">[</span><span class="mi">7</span><span class="p">:]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logger</span><span class="p">.</span><span class="n">warning</span><span class="p">(</span><span class="s">'Better Fig. Error: img_path should start with either {static}, |filename| or /static'</span><span class="p">)</span>
<span class="c1"># Build the source image filename
</span><span class="n">src</span> <span class="o">=</span> <span class="n">instance</span><span class="p">.</span><span class="n">settings</span><span class="p">[</span><span class="s">'PATH'</span><span class="p">]</span> <span class="o">+</span> <span class="n">img_path</span> <span class="o">+</span> <span class="s">'/'</span> <span class="o">+</span> <span class="n">img_filename</span>
<span class="n">logger</span><span class="p">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Better Fig. src: %s'</span><span class="p">,</span> <span class="n">src</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="p">(</span><span class="n">path</span><span class="p">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">src</span><span class="p">)</span> <span class="ow">and</span> <span class="n">access</span><span class="p">(</span><span class="n">src</span><span class="p">,</span> <span class="n">R_OK</span><span class="p">)):</span>
<span class="n">logger</span><span class="p">.</span><span class="n">error</span><span class="p">(</span><span class="s">'Better Fig. Error: image not found: {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">src</span><span class="p">))</span></code></pre></div>
<p>This code basically strips the leading <code>{static}</code>, <code>|filename|</code> or <code>/static</code> from the image path, then looks for the original source image inside the current content folder (as set by the <code>PATH</code> setting in your config). This new code also contains lots more logging for debugging and reporting any errors or warnings. You can see the complete Git commit for the <a href="https://github.com/dflock/pelican-plugins/commit/259147e4da6474c128c4dd09c3a51c64453343af">plugin changes here</a> and the <a href="https://duncanlock.net/blog/2013/05/29/better-figures-images-plugin-for-pelican/">full article on the plugin here</a>.</p></section>Obrigado Deny!2013-09-13T15:32:56-07:002021-06-11T11:09:20-07:00Duncan Locktag:duncanlock.net,2013-09-13:/blog/2013/09/13/obrigado-deny/<div class="image-block"><img src="https://duncanlock.net/images/posts/thanks-deny/Flag_of_Brazil.png" alt="Flag of Brazil"></div>
<p>I’ve recently been corresponding with <a href="http://mexapi.macpress.com.br/">Denny Dias</a>, from Brazil, who’s <a href="http://mexapi.macpress.com.br/2013/08/migrei-meu-blog-do-insosso-blogger-para-a-poderosa-dupla.html">converted his blog over to Pelican</a> - and we’ve been helping each other out a bit with building themes and whatnot.</p>
<p>He’s written up his conversion <span class="amp">&</span> theme building process - and was <a href="http://mexapi.macpress.com.br/2013/09/migrei-meu-blog-customizando-o-tema.html">generous enough to credit me</a> after he borrowed some of my theme’s logic, from this blog’s <a href="https://github.com/dflock/duncanlock.net">GitHub repo</a>.</p>
<p>As he’s such a nice guy - and as I’ve just borrowed his <code>tagsort</code> jinja filter for my <a href="/tags.html">blog’s tag page</a>, I though I’d return the favour - cheers Deny! :)</p>Comprehensive Linux Backups with etckeeper & backupninja2013-08-27T12:36:36-07:002021-02-17T23:50:30-08:00Duncan Locktag:duncanlock.net,2013-08-27:/blog/2013/08/27/comprehensive-linux-backups-with-etckeeper-backupninja/<section id="preamble" aria-label="Preamble"><p>I’m going to build on <a href="http://www.jwz.org/doc/backups.html">Jamie Zawinski’s excellent advice about backups</a>, which you should read first. This is basically that, but with some extra bits. If this seems too complex, then <em>just do what he says</em>.</p>
<p>The plan is to use <a href="https://0xacab.org/liberate/backupninja">Backupninja</a> to backup everything to an external <span class="caps">USB</span> drive – and also to <a href="http://aws.amazon.com/s3/">Amazon S3</a> or <a href="https://www.dropbox.com/">Dropbox</a>, depending on what it is. Backupninja provides a centralized way to configure and schedule many different backup utilities, just by dropping a few simple configuration files into <code>/etc/backup.d/</code>.</p>
<p>I have a multiple hard disk setup for my desktop Linux box - my <code>/home</code> folders live on one disk and <code>/</code> lives on another one. I don’t want to backup everything from the system disk - I can re-install it in 10 mins, and I don’t really want to complicate this …</p></section><section id="preamble" aria-label="Preamble"><p>I’m going to build on <a href="http://www.jwz.org/doc/backups.html">Jamie Zawinski’s excellent advice about backups</a>, which you should read first. This is basically that, but with some extra bits. If this seems too complex, then <em>just do what he says</em>.</p>
<p>The plan is to use <a href="https://0xacab.org/liberate/backupninja">Backupninja</a> to backup everything to an external <span class="caps">USB</span> drive – and also to <a href="http://aws.amazon.com/s3/">Amazon S3</a> or <a href="https://www.dropbox.com/">Dropbox</a>, depending on what it is. Backupninja provides a centralized way to configure and schedule many different backup utilities, just by dropping a few simple configuration files into <code>/etc/backup.d/</code>.</p>
<p>I have a multiple hard disk setup for my desktop Linux box - my <code>/home</code> folders live on one disk and <code>/</code> lives on another one. I don’t want to backup everything from the system disk - I can re-install it in 10 mins, and I don’t really want to complicate this by backing up non-essential stuff. I just want to backup a few system wide configuration items from <code>/</code> - and my MySQL databases, which are kept on there.</p>
<p>To do this, I’m going to tell backupninja to backup the system config, MySQL databases and anything else I want backed up, to my <code>/home</code> folder, then backup the whole <code>/home</code> folder.</p>
<p>I’m also going to do extra backups to Amazon S3, so we’ll have some extra cloud backups of critical stuff.</p>
<p>I’m going to use etckeeper to store the system wide config from <code>/etc</code> in a git repository - this way I get a history of changes and the ability to roll back config changes. I’m then just going to backup that git repository.</p></section>
<section class="doc-section level-1"><h2 id="_making_sure_your_usb_disk_is_mounted_at_backup_time">Making sure your <span class="caps">USB</span> disk is mounted at backup time</h2><p>If you just plug in a <span class="caps">USB</span> drive, it will generally auto-mount and appear inside <code>/media</code>. This is often good enough, but if your machine isn’t setup to do this, or it doesn’t work properly for some reason, you will need to mount the drive permanently by editing <code>/etc/fstab</code>.</p>
<p>I suggest that you mount the backup drive via it’s label, as the device name can change for external devices depending on what’s plugged in at the time. To have an external <span class="caps">USB</span> drive called ‘backups’ and formatted as ext4, mounted at <code>/mnt/backups</code>, first do the following:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo mkdir</span> /mnt/backups</code></pre></div>
<p>then add something like this to your <code>/etc/fstab</code> file:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="ini"><span class="c"># <file system> <mount point> <fs-type> <options> <dump-freq> <pass-num>
</span><span class="py">LABEL</span><span class="p">=</span><span class="s">backups /mnt/backups ext4 nodev,nosuid,noatime,nodiratime 0 0</span></code></pre></div>
<p>you can test this by running:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>mount <span class="nt">-a</span></code></pre></div>
<p>If this works, it shouldn’t print out any errors and browsing to <code>/mnt/backups</code> should show you the contents of your external drive.</p>
<p>In the steps below, I’m going to use <code>/mnt/backups</code> as the backup location, but change this to point to the right place for your local setup.</p></section>
<section class="doc-section level-1"><h2 id="_backing_up_system_configuration_using_etckeeper">Backing up System Configuration using etckeeper</h2><p>Basically, <a href="http://joeyh.name/code/etckeeper/">etckeeper</a> runs itself - you just have to install it and switch it on. You can commit changes to <code>/etc</code> manually if you want to, but by default it hooks into apt to commit changes when you install things and has a cron job to backup outstanding changes daily:</p>
<div class="quote-block"><blockquote><p><em>etckeeper</em> allows the contents of <code>/etc</code> be easily stored in Version Control System (<span class="caps">VCS</span>) repository. It hooks into <em>apt</em> to automatically commit changes to <code>/etc</code> when packages are installed or upgraded. Placing <code>/etc</code> under version control is considered an industry best practice, and the goal of <em>etckeeper</em> is to make this process as painless as possible.</p>
<p> – <a href="https://help.ubuntu.com/12.10/serverguide/etckeeper.html">Ubuntu Server Guide</a></p></blockquote></div>
<p>To install etckeeper, run this in a console:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>git etckeeper</code></pre></div>
<p>The etckeeper from the Ubuntu repositories is setup to use <code>bzr</code> by default, because they’re idiots, so lets change it to use <code>git</code>. Edit the <code>/etc/etckeeper/etckeeper.conf</code> file like so:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="ini"><span class="c"># The VCS to use.
#VCS="hg"
</span><span class="py">VCS</span><span class="p">=</span><span class="s">"git"</span>
<span class="c">#VCS="bzr"
</span><span class="err">#</span><span class="py">VCS</span><span class="p">=</span><span class="s">"darcs"</span></code></pre></div>
<p>If you have bzr installed for some reason, then the etckeeper bzr repository will be automatically initialized. To undo that, run this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>etckeeper uninit</code></pre></div>
<p>Then to re-initialize with a git repository:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>etckeeper init</code></pre></div>
<p>If you don’t have bzr installed it will fail to initialize the bzr repo, so you can just run the second one.</p>
<p>One thing to know about running etckeeper is that it keeps its git repo inside <code>/etc</code> (which is fine) - but this means it runs as root - which takes a bit of getting used to if you’re going to use it manually. You will also need to setup at least a minimal git config for the root user:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>su -
<span class="gp">$</span><span class="w"> </span>git config <span class="nt">--global</span> user.name <span class="s2">"Your Name"</span>
<span class="gp">$</span><span class="w"> </span>git config <span class="nt">--global</span> user.email duncan.lock@gmail.com
<span class="gp">$</span><span class="w"> </span><span class="nb">exit</span></code></pre></div>
<p>Once you’ve done that you can check everything in:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">cd</span> /etc
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>git status
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>etckeeper commit <span class="s2">"Initial Commit"</span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_setting_up_backupninja_postfix">Setting up backupninja <span class="amp">&</span> postfix</h2><p>The Ubuntu package for backupninja also installs<a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a> <code>postfix</code> - which it can use to send notification emails. Postfix is a fully capable - but very lightweight - <span class="caps">MTA</span>/email server. I’m just going to configure it to send outgoing emails and nothing else. This will allow backupninja to send me emails when backups succeed - or fail.</p>
<p>Install backupninja like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>backupninja</code></pre></div>
<section class="doc-section level-2"><h3 id="_configure_postfix">Configure Postfix</h3><p>During install you will see the postfix install wizard, which will prompt you for some configuration values. You can accept the defaults for everything, except these:</p>
<div class="ulist"><ul><li>For ‘type of mail configuration’ select ‘Satellite system’.</li><li>For ‘system mail name’, either accept the default, or enter the domain name to use in the from: address for outgoing emails.</li><li>For ‘relay host’, make sure it’s blank.</li></ul></div>
<p>That should be all the configuration postfix requires. Once the install has completed, you can test it by running this at the command line:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">echo</span> <span class="s1">'test email body'</span> | mail <span class="nt">-s</span> <span class="s1">'test email subject line'</span> send.to.address@wherever.net</code></pre></div>
<p>You should receive an email at <code>send.to.address@wherever.net</code> - remember to check your spam/junk folder. Waking up to an email like this is very reassuring:</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/comprehensive-linux-backups-with-etckeeper-backupninja/backupninja-email-report-screenshot.png" alt="You might have to create a filter or add the from address to your contacts to stop these getting marked as spam."></div></section>
<section class="doc-section level-2"><h3 id="_configure_backupninja">Configure Backupninja</h3><p>The backupninja install will create a config folder: <code>/etc/backup.d</code> where we’ll be storing our backup jobs - and a config file <code>/etc/backupninja.conf</code> which we’ll configure like this - everything else can stay at its defaults:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="ini"><span class="py">reportdirectory</span> <span class="p">=</span> <span class="s">/home/duncan/Dropbox/backups</span>
<span class="py">when</span> <span class="p">=</span> <span class="s">everyday at 02:00</span>
<span class="py">reportemail</span> <span class="p">=</span> <span class="s">your.email@example.net</span></code></pre></div>
<p>I’m sending the backup report log to Dropbox <span class="amp">&</span> email - and kicking everything off at 2am.</p>
<p>The backupninja config files are <em>extremely</em> well commented, explaining what everything does in great detail. The best way to learn how to configure it is just to read the config files. It also installs some thoroughly commented example backup jobs - one of each type - into <code>/usr/share/doc/backupninja/examples/</code> which you can use as the basis for your backup jobs.</p>
<p>Now we’ll setup each of the backup jobs we want to run, by adding a simple text file to the <code>/etc/backup.d</code> folder for each job. These are executed in alphanumeric order, so I suggest you create them like this:</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/comprehensive-linux-backups-with-etckeeper-backupninja/backupninja-etc-backupsd-files.png" alt="Not sure why Thunar thinks that's a Matlab file."></div>
<p>The only caveat is that Backupninja config files need to be owned by root and not world or group readable, so make sure they’re: <code>-rw-------</code>, by doing this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>find /etc/backup.d/ <span class="nt">-type</span> f <span class="nt">-exec</span> <span class="nb">chmod </span>600 <span class="o">{}</span> <span class="se">\;</span></code></pre></div>
<p>and this, to check it worked:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo ls</span> <span class="nt">-lah</span> /etc/backup.d/
<span class="go">
total 40K
drwxrwx--- 2 root root 4.0K May 19 16:54 .
drwxr-xr-x 154 root root 12K May 19 15:25 ..
-rw------- 1 root root 1.4K May 19 16:54 10-little-things.sh
-rw------- 1 root root 3.5K May 19 16:54 50-daily-all-db.mysql
-rw------- 1 root root 219 May 19 16:54 60-daily-home-rsync.sh
-rw------- 1 root root 135 May 19 16:54 70-photos-to-s3.sh
-rw------- 1 root root 134 May 19 16:54 71-ebooks-to-s3.sh
-rw------- 1 root root 138 May 19 16:54 99-cleanup-afterwards.sh</span></code></pre></div>
<p>Speaking of which, backupninja also <em>runs</em> as root, so any files it creates during the backup will be <em>owned</em> by root, so my housekeeping scripts fix that up afterwards.</p></section>
<section class="doc-section level-2"><h3 id="_10_little_things_sh">10-little-things.sh</h3><p>This does some initial housekeeping and copies some little things into the <code>/home</code> folder for later backing up:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># Backup installed packages list</span>
dpkg <span class="nt">--get-selections</span> <span class="o">></span> /home/duncan/backups/dpkg-selections.txt
<span class="c"># Take simple copies of major config files for convenience</span>
<span class="nb">cp</span> /etc/hosts /home/duncan/backups/
<span class="nb">cp</span> /etc/fstab /home/duncan/backups/
<span class="nb">cp</span> /home/duncan/.bashrc /home/duncan/backups/
<span class="nb">cp</span> /home/duncan/.bash_aliases /home/duncan/backups/
<span class="nb">cp</span> /home/duncan/.inputrc /home/duncan/backups/
<span class="nb">cp</span> /home/duncan/.gitconfig /home/duncan/backups/
<span class="nb">cp</span> /home/duncan/.filezilla/sitemanager.xml /home/duncan/backups/
<span class="c"># Copy a few things over to dropbox, for extra safety</span>
<span class="nb">cp</span> /home/duncan/backups/hosts /home/duncan/Dropbox/backups/
<span class="nb">cp</span> /home/duncan/backups/fstab /home/duncan/Dropbox/backups/
<span class="nb">cp</span> /home/duncan/.bashrc /home/duncan/Dropbox/backups/
<span class="nb">cp</span> /home/duncan/.bash_aliases /home/duncan/Dropbox/backups/
<span class="nb">cp</span> /home/duncan/.inputrc /home/duncan/Dropbox/backups/
<span class="nb">cp</span> /home/duncan/.gitconfig /home/duncan/Dropbox/backups/
<span class="nb">cp</span> /home/duncan/.filezilla/sitemanager.xml /home/duncan/Dropbox/backups/
<span class="c"># Backup etckeeper, plus any other git repo's I've backed up to /home/duncan/backups/git-backups</span>
<span class="nb">cd</span> /etc/
git bundle create /home/duncan/backups/git-backups/etc.git-bundle <span class="nt">--all</span>
rsync <span class="nt">-vaxAX</span> <span class="nt">--delete</span> <span class="nt">--ignore-errors</span> /home/duncan/backups/git-backups /home/duncan/Dropbox/backups/git-backups
<span class="c"># Change permissions on the backup folders so that I can use them</span>
<span class="nb">chown</span> <span class="nt">-R</span> duncan /home/duncan/backups/
<span class="nb">chown</span> <span class="nt">-R</span> duncan /home/duncan/Dropbox/backups/</code></pre></div></section>
<section class="doc-section level-2"><h3 id="_50_daily_all_db_mysql">50-daily-all-db.mysql</h3><p>This backs up all my MySQL databases into my home folder using mysqldump:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="ini"><span class="c">### backupninja mysql config file ###
</span>
<span class="py">databases</span> <span class="p">=</span> <span class="s">all</span>
<span class="py">backupdir</span> <span class="p">=</span> <span class="s">/home/duncan/backups/mysql</span>
<span class="py">hotcopy</span> <span class="p">=</span> <span class="s">no</span>
<span class="py">sqldump</span> <span class="p">=</span> <span class="s">yes</span>
<span class="py">compress</span> <span class="p">=</span> <span class="s">yes</span>
<span class="py">dbusername</span> <span class="p">=</span> <span class="s">******</span>
<span class="py">dbpassword</span> <span class="p">=</span> <span class="s">******</span></code></pre></div>
<p>This uses backupninja’s built in support for backing up MySQL databases, so you just need a config file, ending in <code>.mysql</code>, telling it what to backup.</p></section>
<section class="doc-section level-2"><h3 id="_60_daily_home_rsync_sh">60-daily-home-rsync.sh</h3><p>This is the big one that backs up the <code>/home</code> folders to an external <span class="caps">USB</span> disk, provided it’s mounted where it’s supposed to be:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="k">if </span>mountpoint <span class="nt">-q</span> /mnt/backups
<span class="k">then
</span>info <span class="s2">"backup drive is mounted, backing up"</span>
rsync <span class="nt">-vaxAX</span> <span class="nt">--progress</span> <span class="nt">--delete</span> <span class="nt">--ignore-errors</span> <span class="nt">--exclude</span> <span class="s1">'.cache/'</span> <span class="nt">--exclude</span> <span class="s1">'.local/share/Trash/'</span> /home/ /mnt/backups/
<span class="k">else
</span>fatal <span class="s2">"backup drive is not mounted, quitting"</span>
<span class="k">fi</span></code></pre></div>
<p>Backupninja does have support for running rsync backups directly, just like it does for MySQL, but it does time machine style incremental/ hardlink based backups, which wasn’t what I wanted at the moment. I just used this shell script to run rsync - which works fine.</p></section>
<section class="doc-section level-2"><h3 id="_70_photos_to_s3_sh">70-photos-to-s3.sh</h3><p>This one backs up my photo’s to Amazon S3. It requires <a href="http://s3tools.org/s3cmd">s3cmd</a> to be installed and configured:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># Backup photos to Amazon S3</span>
s3cmd <span class="nt">-vH</span> <span class="nt">--progress</span> <span class="nt">--guess-mime-type</span> <span class="nb">sync</span> /home/duncan/Photos/ s3://dflock-backups/dunc-desktop/photos/</code></pre></div>
<p>To install and configure s3cmd, do this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>s3cmd python-magic
<span class="gp">$</span><span class="w"> </span>s3cmd <span class="nt">--configure</span></code></pre></div>
<p>See here for more info on setting up s3cmd:</p>
<div class="quote-block"><blockquote><p>You will be asked for the two keys - copy and paste them from your confirmation email or from your Amazon account page.</p>
<p> – <a class="bare" href="http://s3tools.org/s3cmd">http://s3tools.org/s3cmd</a></p></blockquote></div></section>
<section class="doc-section level-2"><h3 id="_71_ebooks_to_s3_sh">71-ebooks-to-s3.sh</h3><p>I also do the same with my eBooks collection:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># Backup ebooks's to Amazon S3</span>
s3cmd <span class="nt">-vH</span> <span class="nt">--progress</span> <span class="nt">--guess-mime-type</span> <span class="nb">sync</span> /home/duncan/Books/ s3://dflock-backups/dunc-desktop/books/</code></pre></div></section>
<section class="doc-section level-2"><h3 id="_99_cleanup_afterwards_sh">99-cleanup-afterwards.sh</h3><p>This one just does a tiny bit of housekeeping at the end:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># Change permissions on backups so that I can use them</span>
<span class="nb">chown</span> <span class="nt">-R</span> duncan /home/duncan/backups/
<span class="nb">chown</span> <span class="nt">-R</span> duncan /home/duncan/Dropbox/backups/</code></pre></div></section></section>
<section class="doc-section level-1"><h2 id="_testing_with_ninjahelper">Testing with ninjahelper</h2><p>Backupninja comes with a great little tool called <code>ninjahelper</code> to test your backup configurations and manually run jobs. When it starts it gives you a list of each of your jobs:</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/comprehensive-linux-backups-with-etckeeper-backupninja/backupninja-ninjahelper-screenshot.png" alt="backupninja ninjahelper screenshot"></div>
<p>Choose the job you want to test, then you’ll see this:</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/comprehensive-linux-backups-with-etckeeper-backupninja/backupninja-ninjahelper-screenshot-job.png" alt="backupninja ninjahelper screenshot job"></div>
<p>Do a test run, then a real run of each job. This will also test permissions etc…​ and tell you if anything needs changing.</p>
<p>Use this to do a test run of each of your jobs in turn until they work, then to actually run each one and check the output. Once they all work here, you’re good to go.</p>
<p>You can check your backup system configuration changes into <code>etckeeper</code> now:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>etckeeper commit <span class="s2">"Initial setup of backup system"</span></code></pre></div>
<p>So, your backup system configuration is now backed up :)</p></section>
<section class="doc-section level-1"><h2 id="_physical_off_site_backups">Physical Off-site Backups</h2><p>I also want <em>physical</em> off-site backups of everything - in case anything happens to my building - like a fire, flood or burglary, for example.</p>
<p>Once you’ve setup the above, this is simplicity itself - just remove the external <span class="caps">USB</span> backup disk, stick a post-it note with the date on it, and take it to work, or give it to a friend who lives separately from you.</p>
<p>Then just get a new blank disk and put it where the old one was, format, label and mount it the same way. Backups will then happen to that disk.</p>
<p>Then, like <a href="http://www.jwz.org/doc/backups.html">jwz</a> says - every month, bring that other drive back, plug it in and run the backup to it, then either take it away again or swap them over.</p>
<section class="doc-section level-2"><h3 id="_mounting_unmounting">Mounting <span class="amp">&</span> Unmounting</h3><p>To un-mount your existing backup disk, so you can safely remove it, do this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>umount /mnt/backups</code></pre></div>
<p>Then remove it and plug the new disk in. Make sure it’s formatted and labeled correctly<a class="footnote-ref" id="_footnoteref_2" href="#_footnote_2" title="View footnote 2" role="doc-noteref">[2]</a>, then mount it like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>mount <span class="nt">-a</span></code></pre></div>
<p>Which will mount everything in your <code>/etc/fstab</code> that isn’t already mounted.</p></section></section>
<section class="doc-section level-1"><h2 id="_testing">Testing</h2><p>I’m deliberately not doing anything too fancy here - no compression, no encryption, etc…​ - just a simple copy of stuff. This means less things to go wrong - and that testing is easier. Open some files from the backup and check that they’re <span class="caps">OK</span>.</p>
<p>Copy some files off the backup disk to check that works; download some stuff from s3.</p>
<p>Do this periodically. Backups that don’t restore are worse than no backups.</p>
<p>Remember to keep an eye on the log file that Backupninja makes at <code>/var/log/backupninja.log</code> - and make sure you’re getting the emails - and check and immediately fix any errors or failures.</p></section>
<section class="doc-section level-1"><h2 id="_then_relax">Then…​ relax</h2><p>Once this is all setup, you can take a deep breath and relax - safe in the knowledge that you’re covered if anything bad happens to your digital life. This only took me a couple of hours to setup from scratch - but will take you much less because you can copy <span class="amp">&</span> paste my hard work. What are you waiting for - give yourself the gift of some peace of mind.</p>
<hr>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References</h3></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">The Ubuntu packages install <a href="http://www.postfix.org/">postfix</a> when you install backupninja via a dependency on <code>+mail-utils+</code>, which depends on <code>+mail-transport-agent+</code>, which is provided by <code>+postfix+</code>. <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">i.e. usually the same format as your source drive (ext4 in my case) and labeled ‘backups’. I use the excellent <a href="http://gparted.sourceforge.net/">GParted</a> for this, which you can install from your distributions repository in the usual way. <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>How to convert FLAC files from 24/48 bit to 16 bit on Ubuntu Linux2013-08-19T22:40:59-07:002021-06-11T11:07:44-07:00Duncan Locktag:duncanlock.net,2013-08-19:/blog/2013/08/19/how-to-convert-flac-files-from-24-48-bit-to-16-bit-on-ubuntu-linux/<p>I recently needed to convert some <span class="caps">FLAC</span> music files from the increasingly common 48 bit encoding, down to 16 bit at 44100 kHz, so that they’ll play on my Sonos. Here’s how to do it:</p>
<p>If you don’t already have <code>sox</code> installed, do this to install it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>sox</code></pre></div>
<p>Then run this to do the conversion, in the folder with music in:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">mkdir </span>resampled
<span class="gp">$</span><span class="w"> </span><span class="k">for </span>flac <span class="k">in</span> <span class="k">*</span>.flac<span class="p">;</span> <span class="k">do </span>sox <span class="nt">-S</span> <span class="s2">"</span><span class="k">${</span><span class="nv">flac</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-r</span> 44100 <span class="nt">-b</span> 16 ./resampled/<span class="s2">"</span><span class="k">${</span><span class="nv">flac</span><span class="k">}</span><span class="s2">"</span><span class="p">;</span> <span class="k">done</span></code></pre></div>
<p>And that’s it - it will convert all the <code>.flac</code> files in that folder to 16 bit at 44100 kHz and put the result into the <code>./resampled</code> subfolder, preserving the metadata.</p>Sunshine Coast: Thormanby Islands & Smuggler Cove Kayak Trip2013-07-17T00:05:32-07:002021-05-27T09:02:55-07:00Duncan Locktag:duncanlock.net,2013-07-17:/blog/2013/07/17/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/<p>A trip report from our Canada Day 2013 long weekend kayak camping trip to Thormanby Islands, Smuggler Cove <span class="amp">&</span> Secret Cove, on the sunshine coast.</p>
<figure class="image-block"><img alt="thormanby trip map" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/thormanby-trip-map.png"/>
<figcaption>Figure 1. We started at Welcome Beach on Saturday afternoon, kayaked over to Thormanby for lunch, then up and round to Buccaneer Bay - stunning camp on the dunes, then over to Smugglers Cove on Sunday. Camped there Sunday night, with a trip to Secret Cove in the afternoon, then back to Welcome Beach on Monday morning. (Map screenshot courtesy and copyright Google Maps. Annotations added by me.)</figcaption></figure>
<section class="doc-section level-1"><h2 id="_welcome_beach_to_thormanby">Welcome Beach to Thormanby</h2><p>On Saturday morning, we packed up our tent <span class="amp">&</span> got picked up by Erika from Talaysay Tours, who drove us, with the Kayak on the roof, to Welcome Beach, about 15 mins further up Highway 101, past Sechelt.</p>
<p>We unloaded the Kayak, took it down to …</p></section><p>A trip report from our Canada Day 2013 long weekend kayak camping trip to Thormanby Islands, Smuggler Cove <span class="amp">&</span> Secret Cove, on the sunshine coast.</p>
<figure class="image-block"><img alt="thormanby trip map" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/thormanby-trip-map.png"/>
<figcaption>Figure 1. We started at Welcome Beach on Saturday afternoon, kayaked over to Thormanby for lunch, then up and round to Buccaneer Bay - stunning camp on the dunes, then over to Smugglers Cove on Sunday. Camped there Sunday night, with a trip to Secret Cove in the afternoon, then back to Welcome Beach on Monday morning. (Map screenshot courtesy and copyright Google Maps. Annotations added by me.)</figcaption></figure>
<section class="doc-section level-1"><h2 id="_welcome_beach_to_thormanby">Welcome Beach to Thormanby</h2><p>On Saturday morning, we packed up our tent <span class="amp">&</span> got picked up by Erika from Talaysay Tours, who drove us, with the Kayak on the roof, to Welcome Beach, about 15 mins further up Highway 101, past Sechelt.</p>
<p>We unloaded the Kayak, took it down to the surf and loaded all our gear into it, ready to paddle off into the blue.</p>
<figure class="image-block"><a class="image" href="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060379.jpg"><img alt="p1060379 small" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060379-small.jpg"/></a>
<figcaption>Figure 2. Standing on Welcome Beach, looking south-west across Welcome Passage towards the gap between Merry Island on the left <span class="amp">&</span> South Thormanby Island on the right.</figcaption></figure>
<p>The first stretch of paddling was about 45 minutes, across the widest part of Welcome Passage. The sea here is very sheltered and calm - and we had stunning weather: hot and sunny, with just enough sea breeze to keep cool. Welcome Passage is quite busy with boat traffic and the water was very murky, full of churned up algae, sea weed and a fair bit of floating lumber. We didn’t spot any marine mammals in Welcome Passage the entire weekend - and not much other sea-life either. It’s a busy <span class="amp">&</span> noisy waterway in the summer (by <span class="caps">BC</span> standards) - a huge contrast to the relatively pristine waters of Sechelt Inlet, where Seals, Starfish, Sunstars and Sea Cucumbers abound.</p>
<p>Once we’d made it to the other side, we pulled up at a tiny deserted beach about ½ way up the east side of South Thormanby Island, for cider and snacks. The water here was a bit clearer - and we watched Ospreys hunt in the bay in front of us while we rested:</p>
<figure class="image-block"><a class="image" href="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060391.jpg"><img alt="p1060391 small" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060391-small.jpg"/></a>
<figcaption>Figure 3. You can see Welcome Beach where we started, in the background, back across Welcome Passage.</figcaption></figure></section>
<section class="doc-section level-1"><h2 id="_welcome_passage_to_buccaneer_bay">Welcome Passage to Buccaneer Bay</h2><div class="image-block"><img alt="welcome passage to buccaneer bay" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/welcome-passage-to-buccaneer-bay.png"/></div>
<p>After a quick break we paddled off, up the east side of the island, round the top and into Buccaneer Bay. As Welcome Passage narrows the current and swell get slightly stronger - but it was such a calm day it was hardly noticeable. This side of Thormanby is very craggy with step cliffs and deep little bays - topped with characteristically gnarly Arbutus trees.</p>
<p>All along the passage, the high tide line was strongly marked on the cliffs at each side - visible even on the opposite side. This turned out to be a carpet of tiny mussels, almost like fur, clinging to the rocks <span class="amp">&</span> completely covering them from the high-water mark down.</p>
<p>Once you reach the top of the passage, the big blue expanse of the Georgia Strait opens in front of you, with the Sunshine Coast on the right and Texada Island in the distance on the left.</p>
<p>The top of South Thormanby ends in a series of craggy fingers of rock, sloping down into the Strait, each with a long deep bay between. Some of these have small houses and private docks in - with, presumably, fabulous views up the coast.</p>
<p>Once you round the corner, the very large sheltered Buccaneer Bay opens up on your left. There were lots of boats moored in and around the islets at the top of the bay, and a few holiday cottages and docks along the shore. These gave way to larger boats as we neared the anchorage at the end of the bay.</p>
<p class="jpg"><img alt="p1060403 small" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060403-small.jpg"/></p></section>
<section class="doc-section level-1"><h2 id="_camping_at_the_gap">Camping at the Gap</h2><p>We beached the Kayak at the end of the bay, at <a href="http://www.env.gov.bc.ca/bcparks/explore/parkpgs/buccaneer_bay/">Buccaneer Bay Provincial Park</a>. This is a sand spit that joins North and South Thormanby Islands together - with space for ~10 tents. It has stunning views - if you face one way, you can see up Buccaneer Bay, to the Sunshine Coast, with the Coastal Mountains in the distance - if you turn around, you can see across Georgia Strait to Vancouver Island on the horizon. Welcome to The Gap:</p>
<figure class="image-block jpg"><img alt="link:{static}/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060430" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060430-small.jpg"/>
<figcaption>Figure 4. Standing on the Gap, looking north east, towards the mouth of the bay.</figcaption></figure>
<figure class="image-block jpg"><img alt="link:{static}/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060428" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060428-small.jpg"/>
<figcaption>Figure 5. …and looking south west, towards Vancouver Island.</figcaption></figure>
<p>You could easily imagine 17th century pirates anchoring their galleons in the bay and drinking rum around campfires on the beach - an effect heightened when this two masted schooner sailed into the bay a few hours after we arrived:</p>
<figure class="image-block"><a class="image" href="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060467.jpg"><img alt="p1060467 small crop" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060467-small-crop.jpg"/></a>
<figcaption>Figure 6. This beautiful two masted schooner sailed into the bay a few hours after we arrived. As we sat on the beach, reading, propped up on a log, they dropped sail, anchored and the crew rowed ashore in long boats.</figcaption></figure>
<p>We made camp, walked around the beaches, took in the views and had a relaxed dinner. At some point, our neighbours slipped off in one of their kayaks. They returned about an hour later, with the front cockpit full of fish - which they cleaned and gutted in the surf and cooked over a fire. Prior to this, I was quite happy with my stew, but I admit, I was rather envious of their fishing skills.</p>
<p>We were treated to a glorious sunset to round off a pretty perfect day:</p>
<div class="image-block"><a class="image" href="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060487.jpg"><img alt="p1060487 small" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060487-small.jpg"/></a></div></section>
<section class="doc-section level-1"><h2 id="_buccaneer_bay_to_smuggler_cove">Buccaneer Bay to Smuggler Cove</h2><div class="image-block"><img alt="buccaneer bay to smugglers cove" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/buccaneer-bay-to-smugglers-cove.png"/></div>
<p>On Sunday morning, we had a lazy brunch on the beach - and watched an Osprey catch theirs in the bay, right in front of us. We eventually packed the camp back into the kayak and headed out, up the east side of North Thormanby island, to check out the beaches we saw on the way in.</p>
<p>There are lots more holiday cottages and lots more boats moored on this side of the bay, especially as you approach the long, sandy, Vaucroft Beach. This is a popular day trip spot for boaters, as it’s very close to Secret Cove, a fairly large marina. The beach itself is light sand, fairly unusual in <span class="caps">BC</span>, and could easily pass for the Caribbean, on a sunny day.</p>
<figure class="image-block"><a class="image" href="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060490.jpg"><img alt="p1060490 small" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060490-small.jpg"/></a>
<figcaption>Figure 7. There was also a float plane moored up just off Vaucroft Beach.</figcaption></figure>
<p>After passing Vaucroft beach, we headed east, towards the coast and Smuggler Cove. We paddled across the wide mouth of Buccaneer Bay, past the fingers of South Thormanby and across Welcome Passage in the <em>glorious</em> sunshine. Paddling towards the coast, we had great views of the Coastal Mountains ahead of us and the expanse of the Strait on our left the whole way over. We passed quite a few boats buzzing back and forth between Secret Cove <span class="amp">&</span> Thormanby, including a water taxi.</p>
<p>The entrance to <a href="http://www.env.gov.bc.ca/bcparks/explore/parkpgs/smuggler/">Smuggler Cove</a> is almost invisible until you get quite close. A small gap between two rocky outcrops, leads you into a little hidden world of sheltered bays and waterways, almost completely separate from the ocean outside. Just inside the entrance in the first large cove, we came across 16 yachts moored up and rafted together - “Millionaires Row”. We later met a very nice couple from one of these yachts - apparently it was a local club meeting up for the weekend.</p>
<figure class="image-block"><a class="image" href="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/smuggler-cove-millionaires-row-panorama.jpg"><img alt="A panorama of row of yachts rafted up for the weekend at Smuggler Cove." src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/smuggler-cove-millionaires-row-panorama-small.jpg"/></a>
<figcaption>Figure 8. A panorama of row of yachts rafted up for the weekend at Smuggler Cove. The gap on the left is the entrance to the cove, with the tip of North Thormanby island and Vaucroft Beach visible in the distance</figcaption></figure>
<p>We wound our way deeper into the cove, around islands and moored boats, looking for somewhere to beach - or for signs of the camp site. Eventually we found a small muddy beach right at the southern end of the cove and hauled out. We scouted around and found people camping just up in the woods, along with a pit toilet. We unpacked the kayak and made camp.</p>
<figure class="image-block"><a class="image" href="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060524.jpg"><img alt="Our little tent." src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060524-small.jpg"/></a>
<figcaption>Figure 9. Our little tent.</figcaption></figure>
<p>Once we’d got everything set up, we headed back out, for a trip to Secret Cove.</p></section>
<section class="doc-section level-1"><h2 id="_trip_to_secret_cove_for_ice_cream_real_bathroom">Trip to Secret Cove, for Ice Cream, Real Bathroom</h2><p><img alt="trip to secret cove for ice cream" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/trip-to-secret-cove-for-ice-cream.png"/></p>
<p>We paddled back out of the cove and right, around Capri Isle and along the coast towards <a href="https://www.google.ca/maps?t=m&ll=49.52996589999999%2C-123.96011350000003&spn=0.026796243931675245%2C0.05504765799669358">Secret Cove</a>. True to it’s name, Secret Cove is well hidden behind islands and it opens up into a series of long deep bays once you get past the entrance - most of it isn’t visible until you get inside, past Jack Tolmie and Turnagain Islands.</p>
<p>There are several marinas, chandleries, floating bars and tens - possibly hundreds - of docks, moorings and cottages nestled inside Secret Cove’s many deep bays, accommodating hundreds of water craft of all shapes and sizes. From our little kayak, to old log-raft tugs, the occasional trawler, multi-million dollar yachts - and everything in between.</p>
<p>Floating in, under and through this whole… regatta in the sunshine, in our little kayak, was interesting and fun.</p>
<p>Over to the north west side we eventually found the government dock and <a href="http://www.secretcovemarina.com/gallery">marina</a> - with a boat fuelling dock, floating restaurant and shop. We tied the kayak to the dock at the back and climbed out. We proceeded to make <em>extensive</em> use of their bathroom facilities, before buying cold drinks, ice cream and After Sun lotion from the shop. We then sat in the shade on the edge of the dock and watched the world go by for a while.</p></section>
<section class="doc-section level-1"><h2 id="_back_to_smuggler_cove_dinner_beavers">Back to Smuggler Cove, dinner <span class="amp">&</span> Beavers</h2><p>We paddled back to Smuggler Cove, tied up the kayak and explored the trails around the cove a little before dinner.</p>
<p>After dinner, we walked along the forest trail the other way, away from the cove into the woods. A little way in, in the gathering twilight, the trail gave way to a wooden walkway over a shallow lake. The lake turned out to be a wetland habitat created by <a href="http://en.wikipedia.org/wiki/Beaver">beavers</a>, who had drowned the bottom of this little forest valley by damming a creek.</p>
<figure class="image-block"><a class="image" href="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060519.jpg"><img alt="p1060519 small" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/p1060519-small.jpg"/></a>
<figcaption>Figure 10. Our first Beavers! This is looking away from the dam, down the valley. We took this picture in the morning, when we came back with the camera. That walkway is courtesy of Parks Canada, not the beavers, obviously.</figcaption></figure>
<p>We didn’t know this was here, so it was a nice surprise - and the drowned forest was very atmospheric in the gloaming.</p>
<p>As we approached the edge of the lake, we almost immediately disturbed a beaver - alarmed, it made a loud slap with its broad tail on the water’s surface, dove in and swam away.</p>
<p>As we walked around the lake, we heard two more making their (very load) tail slapping alarm call and swimming off. We just saw the tail end of one of them as it dived off a log.</p></section>
<section class="doc-section level-1"><h2 id="_homeward_bound_smuggler_cove_to_welcome_beach">Homeward Bound: Smuggler Cove to Welcome Beach</h2><p>We went and checked out the Beavers again in the morning light - hoping to see one this time, but no luck. The lake was buzzing with dragonflies and we saw a frog (or maybe a toad?) basking on the mud at the side of the walkway - just a few of the <a href="http://en.wikipedia.org/wiki/Beaver_dam#Benefits">benefits of beaver dams</a>.</p>
<p><img alt="smuggler cove to welcome beach" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/smuggler-cove-to-welcome-beach.png"/></p>
<p>We left Smuggler Cove before lunch, because we needed to be back at Welcome Beach by 2pm, to be picked up by Erika for our ride back into Sechelt. The paddle back was about 7 km, which took us roughly 2 hours.</p>
<p>Paddling back down Welcome Passage, we passed several herons, fishing from the rocks by the shore. We also passed a <em>huge</em> yacht, ploughing it’s way north.</p>
<p>We eventually found the correct beach and hauled out, rested, had lunch and unpacked. Erika arrived at 2pm and we loaded the Kayak and our gear into her jeep and she drove us back to Sechelt.</p>
<p>Sechelt were having their Canada day parade, so after a few diversions, we <em>just</em> made it to the Langdale bus in time - for the start of the trip home.</p>
<p>All in all - an amazing weekend. Shower time!</p>
<hr/></section>
<section class="doc-section level-1"><h2 id="_logistics_getting_there_from_vancouver">Logistics <span class="amp">&</span> Getting there from Vancouver</h2><p>We don’t own a car - but getting to the sunshine coast is very easy on public transport - and much cheaper than taking a car on the ferry. Here’s how we did it:</p>
<p>We took the #257 Express Bus from down-town Vancouver to Horseshoe Bay. We normally get on at the stop on West Georgia St., right outside The Bay. You can also catch the #250 from here - which isn’t an express but doesn’t take too much longer. This costs $2.75 per person.</p>
<p>Once we got to Horseshoe Bay, we took the ferry to Langdale. These are fairly frequent, but with occasional gaps, so <a href="http://www.bcferries.com/schedules/mainland/vasc-current.php">check the schedule</a>. As a foot passenger, we’ve never had to wait or not been able to get on - we just walk onto the first ferry that turns up. This is $15 per person, including the return trip. If you’re planning to do this often, it might be worth getting a <a href="https://www.bcferries.com/experience_and_coast_card/"><span class="caps">BC</span> Ferries Experience card</a>. You have to pre-load it with at least $60, but you get ~20% off most<a class="footnote-ref" href="#_footnote_1" id="_footnoteref_1" role="doc-noteref" title="View footnote 1">[1]</a> fares.</p>
<figure class="image-block"><img alt="thormanby trip overview map" src="https://duncanlock.net/images/posts/sunshine-coast-thormanby-islands-smuggler-cove-kayak-trip/thormanby-trip-overview-map.png"/>
<figcaption>Figure 11. Bus from Vancouver to Horseshoe Bay, then Ferry to Langdale, followed by bus to Sechelt. Taxi to Porpoise Bay camp site, stay overnight. Lift to Welcome Beach with Kayak people, then off! Map screenshot courtesy (and copyright) Google Maps. Annotations added by me.</figcaption></figure>
<p>The ferry crossing is a <em>very</em> scenic 45 min trip across Howe Sound. Once we arrived at Langdale, we followed the other foot passengers out, through the foot passenger tunnel to the car parks, then caught the ‘Highway 101’ bus - it’s the only bus from the only bus stop, so you can’t really get this wrong - although there are express and non-express buses, which are quite a bit slower. Anyway, we got off at Sechelt, outside Trail Bay mall (the last stop). This costs $2.25 per person.</p>
<section class="doc-section level-2"><h3 id="_porpoise_bay">Porpoise Bay</h3><p>We stayed overnight at <a href="http://www.env.gov.bc.ca/bcparks/explore/parkpgs/porpoise/">Porpoise Bay Provincial Park</a>, just outside Sechelt. This park - and Sechelt Inlet that it’s on, is worth a trip on its own - the Inlet has very nice sheltered Kayaking with lots of quiet, empty camping and loads of wildlife.</p>
<p>To get here, we normally catch a Taxi from Trail Bay mall in Sechelt to the park - it’s ~5km out of Sechelt; this costs ~$15, call Sechelt Taxi 604-989-8294 – and <a href="http://www.env.gov.bc.ca/bcparks/fees/"><span class="caps">BC</span> Parks are ~$11 per night, per group</a>.</p>
<p>For this trip we overnighted here so that we could get a lift to and from Welcome Beach (and rent a Kayak) from <a href="http://www.talaysay.com/">Talaysay Tours, who rent Kayaks from the beach at Porpoise Bay</a>, among other locations. They have good equipment and are extremely friendly <span class="amp">&</span> helpful.</p>
<hr/></section>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References</h3></section></section><section aria-label="Footnotes" class="footnotes" role="doc-endnotes"><hr/><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote"><span class="caps">BC</span> Ferries Experience Card Summary: You have to pre-load with $60 at a time and you get ~20% off tickets, although not all routes, see <a href="http://www.bcferries.com/experience_and_coast_card/what_it_is/">here</a> for more info. For example, rather cynically, you can pay for a Horseshoe Bay to Nanaimo ticket with one but you don’t get any discount. Also there a load of T&C’s, so <a href="http://www.bcferries.com/experience_and_coast_card/what_it_is/FAQ.html">think about it</a> before getting one. <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>How I compress PNG files on this website2013-07-05T16:02:45-07:002021-06-01T06:26:23-07:00Duncan Locktag:duncanlock.net,2013-07-05:/blog/2013/07/05/how-i-compress-png-files-on-this-website/<section class="doc-section level-1"><h2 id="_compressing_limited_colour_png_images">Compressing Limited Colour <span class="caps">PNG</span> images</h2><p>Most of the .<span class="caps">PNG</span> files on this site are the ‘blueprint’ style diagrams, like this one:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/how-i-compress-png-files-on-this-website/microscope-diagram.png" alt="Blueprint style diagram showing an optical microscope. The sample inside is magnified in a bubble to 2500 times, showing it to be a complex, detailed paisley pattern." width="500">
<figcaption>Figure 1. This illustration is a large 5.<span class="caps">6MB</span> <span class="caps">SVG</span> file, mostly because of the very detailed paisley pattern that I used. Exported to <span class="caps">PNG</span> - and then compressed using the process below, you can get this down to 118.5kB.</figcaption></figure>
<p>I create these in Inkscape as vector .<span class="caps">SVG</span> files <span class="amp">&</span> export them to bitmap .<span class="caps">PNG</span> files. I then re-compress them, to ensure that the image files that are used on the live website are as small <span class="amp">&</span> quick to load as possible.</p>
<p>As these diagrams have a fairly limited colour palette, I can get lots of extra compression by reducing the colour depth of the final .<span class="caps">PNG</span> files from the default 32bit (millions of colours) to 8bit (256 colours) - without any …</p></section><section class="doc-section level-1"><h2 id="_compressing_limited_colour_png_images">Compressing Limited Colour <span class="caps">PNG</span> images</h2><p>Most of the .<span class="caps">PNG</span> files on this site are the ‘blueprint’ style diagrams, like this one:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/how-i-compress-png-files-on-this-website/microscope-diagram.png" alt="Blueprint style diagram showing an optical microscope. The sample inside is magnified in a bubble to 2500 times, showing it to be a complex, detailed paisley pattern." width="500">
<figcaption>Figure 1. This illustration is a large 5.<span class="caps">6MB</span> <span class="caps">SVG</span> file, mostly because of the very detailed paisley pattern that I used. Exported to <span class="caps">PNG</span> - and then compressed using the process below, you can get this down to 118.5kB.</figcaption></figure>
<p>I create these in Inkscape as vector .<span class="caps">SVG</span> files <span class="amp">&</span> export them to bitmap .<span class="caps">PNG</span> files. I then re-compress them, to ensure that the image files that are used on the live website are as small <span class="amp">&</span> quick to load as possible.</p>
<p>As these diagrams have a fairly limited colour palette, I can get lots of extra compression by reducing the colour depth of the final .<span class="caps">PNG</span> files from the default 32bit (millions of colours) to 8bit (256 colours) - without any loss of visual quality.</p>
<p>This is a two-step process - which can be combined into a single pipeline. The first step is <code>pngnq</code> which quantizes the image, reducing its colour depth. The second step uses <code>advpng</code> to losslessly re-compress this image data to get the smallest possible filesize.</p>
<p>To install these two utilities on ubuntu/debian Linux, run this in a console:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>advancecomp pngnq</code></pre></div>
<p>The basic idea is to do the following to the file, running it through both utilities, one after the other:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pngnq <span class="nt">-s1</span> image.png
<span class="gp">$</span><span class="w"> </span>advpng <span class="nt">-z</span> <span class="nt">-4</span> image.png</code></pre></div>
<p>We can do that in one go by joining the commands together, like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pngnq <span class="nt">-s1</span> image.png <span class="o">&&</span> advpng <span class="nt">-z</span> <span class="nt">-4</span> <span class="nt">-q</span> image-nq8.png</code></pre></div>
<p>By default, <code>pngnq</code> adds <code>-nq8</code> onto the end of the filename. You can change that with the <code>-e</code> parameter, but it doesn’t seem to support not changing the name - i.e. overwriting to original. As an added complication <code>advpng</code> doesn’t support processing from stdin to stdout, so we have to pass it a filename.</p>
<p>If we want to process a whole folder full of files, we can do this in parallel, getting all our machines cores in on the action, using <code>find</code> and <code>xargs</code>. This example uses 4 processes to do 4 files at once:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>find <span class="nb">.</span> <span class="nt">-iname</span> <span class="s2">"*png"</span> <span class="nt">-print0</span> | xargs <span class="nt">-r0</span> <span class="nt">--max-procs</span><span class="o">=</span>4 <span class="nt">-n1</span> sh <span class="nt">-c</span> <span class="s1">'pngnq -s1 "$1" && advpng -z -4 -q "${1%.*}"-nq8.png'</span> -</code></pre></div>
<p>You can adjust the <code>--max-procs=4</code> bit to match the number of cores you have - or maybe one less, if you want to process a lot of images and do other things in the meantime.</p></section>
<section class="doc-section level-1"><h2 id="_results">Results</h2><p>Using the microscope image at the start of this post as an example, these are the results that we get at each step:</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 50%;"><col style="width: 50%;"></colgroup><thead><tr><th class="halign-left valign-top">Process Step</th><th class="halign-left valign-top">Result</th></tr></thead><tbody><tr><td class="halign-left valign-top">1: Original <span class="caps">SVG</span> File</td><td class="halign-left valign-top">5.6 <span class="caps">MB</span></td></tr><tr><td class="halign-left valign-top">2: Initial <span class="caps">PNG</span> Export</td><td class="halign-left valign-top">409.3 kB</td></tr><tr><td class="halign-left valign-top">3: Quantized with pngnq</td><td class="halign-left valign-top">123.6 kB</td></tr><tr><td class="halign-left valign-top">4: Compressed with advpng</td><td class="halign-left valign-top">118.5 kB</td></tr></tbody></table></div>
<p>This example is a <em>huge</em> <span class="caps">SVG</span> file by my standards - the paisley vector clipart I used is <em>very</em> detailed and ~<span class="caps">3MB</span> on its own; my normal <span class="caps">SVG</span> files are generally under 1024 kB. If we use a more typical <span class="caps">SVG</span> file, like the one from my <a href="https://duncanlock.net/blog/2013/06/19/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/">Reddit Bots article</a> (which is 835.2 kB), we still get excellent compression, down to 78.5 kB – just 9% of the original size and less than ⅓ of the initial <span class="caps">PNG</span> export size:</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/how-i-compress-png-files-on-this-website/compression-results-diagram.png" alt="Blueprint style diagram showing the relative sizes of each step of the compression process as circles, going from 832.2 Kb to 78.5 kB"></div>
<p>Re-compressing all the blueprint style diagrams on this site - which I had previously <strong>already re-compressed using <span class="caps">PNGOUT</span></strong> - reduced their total file size from 845.9 Kb to 385.2 Kb - a whopping <strong>~55% reduction</strong>. Just goes to show that you need to test and adapt techniques to your particular situation - rather than just applying them blindly.</p></section>
<section class="doc-section level-1"><h2 id="_general_png_compression">General <span class="caps">PNG</span> Compression</h2><p>You don’t always want to reduce the colour palette - not all images are suitable for this. In that case, you can reduce the process to only one step. You can do this, to process a folder full of <span class="caps">PNG</span> files, four at once:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>find <span class="nb">.</span> <span class="nt">-iname</span> <span class="s2">"*png"</span> <span class="nt">-print0</span> | xargs <span class="nt">-0</span> <span class="nt">--max-procs</span><span class="o">=</span>4 <span class="nt">-n</span> 1 advpng <span class="nt">-z</span> <span class="nt">-4</span> <span class="nt">-q</span></code></pre></div>
<p>This just runs each file through <code>advpng</code> to maximally compress the existing image data, losslessly, without changing the colour depth or quality at all.</p>
<p>To make this easier, I created some <a href="https://duncanlock.net/blog/2013/06/28/useful-thunar-custom-actions/">Thunar Custom Actions</a> that give you the option to do either of these things when you right-click on a <span class="caps">PNG</span> file.</p></section>Useful Thunar Custom Actions2013-06-28T14:11:43-07:002021-06-10T21:04:53-07:00Duncan Locktag:duncanlock.net,2013-06-28:/blog/2013/06/28/useful-thunar-custom-actions/<div class="image-block"><img alt="Thunar's icon" height="stylized version of Thor's hammer" src="https://duncanlock.net/images/posts/useful-thunar-custom-actions/thunar-icon.png" width="a beautifully rendered"/></div>
<p>Thunar - <span class="caps">XFCE</span> <span class="amp">&</span> XUbuntu’s small but perfectly formed file manager - has a simple mechanism that allows you to easily add new commands to the right click menu of files and folders. These are called <a href="http://docs.xfce.org/xfce/thunar/custom-actions">Custom Actions</a> and are easy to create… here’s how to do it.</p>
<p>Click the <strong>Edit</strong> menu, then click ‘<strong>Configure custom actions…</strong>‘. This will take you to the Custom Actions Manager, where you can create, edit or delete your custom actions.</p>
<p>You can enter anything into the command box, including complex bash scripts, names of scripts or executables on the <span class="caps">PATH</span>, or the full path and filename of the command you want to run.</p>
<div class="image-block"><img alt="thunar custom actions edit 1" src="https://duncanlock.net/images/posts/useful-thunar-custom-actions/thunar-custom-actions-edit-1.png"/></div>
<p>On the ‘<strong>Appearance Conditions</strong>‘ tab, you tell Thunar when you want your item to appear in the right click menu:</p>
<figure class="image-block"><img alt="thunar custom actions edit 3" src="https://duncanlock.net/images/posts/useful-thunar-custom-actions/thunar-custom-actions-edit-3.png"/>
<figcaption>Figure 1. Now, when I right click on a text file, I …</figcaption></figure><div class="image-block"><img alt="Thunar's icon" height="stylized version of Thor's hammer" src="https://duncanlock.net/images/posts/useful-thunar-custom-actions/thunar-icon.png" width="a beautifully rendered"/></div>
<p>Thunar - <span class="caps">XFCE</span> <span class="amp">&</span> XUbuntu’s small but perfectly formed file manager - has a simple mechanism that allows you to easily add new commands to the right click menu of files and folders. These are called <a href="http://docs.xfce.org/xfce/thunar/custom-actions">Custom Actions</a> and are easy to create… here’s how to do it.</p>
<p>Click the <strong>Edit</strong> menu, then click ‘<strong>Configure custom actions…</strong>‘. This will take you to the Custom Actions Manager, where you can create, edit or delete your custom actions.</p>
<p>You can enter anything into the command box, including complex bash scripts, names of scripts or executables on the <span class="caps">PATH</span>, or the full path and filename of the command you want to run.</p>
<div class="image-block"><img alt="thunar custom actions edit 1" src="https://duncanlock.net/images/posts/useful-thunar-custom-actions/thunar-custom-actions-edit-1.png"/></div>
<p>On the ‘<strong>Appearance Conditions</strong>‘ tab, you tell Thunar when you want your item to appear in the right click menu:</p>
<figure class="image-block"><img alt="thunar custom actions edit 3" src="https://duncanlock.net/images/posts/useful-thunar-custom-actions/thunar-custom-actions-edit-3.png"/>
<figcaption>Figure 1. Now, when I right click on a text file, I have extra options in my menu.</figcaption></figure>
<p>I’ve included my custom actions <a href="#_my_thunar_custom_actions">below</a> - and you can find <a href="https://www.google.com/search?q=thunar+custom+actions">more around the web</a>. I’ve only included ones here that aren’t commonly listed elsewhere.</p>
<section class="doc-section level-1"><h2 id="_but_first_a_word_about_working_directories_variables">But first, a word about Working Directories <span class="amp">&</span> Variables</h2><section class="doc-section level-2"><h3 id="_working_directory">Working Directory</h3><p>The current working directory for the a custom action, is the folder the Thunar window that launched the action, is currently displaying. You can test this by creating and running the following custom action:</p>
<section class="doc-section level-3"><h4 id="_test_cwd">Test <span class="caps">CWD</span></h4><div class="dlist"><dl><dt>Description</dt><dd>Prints out the current working directory</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">pwd</span> | zenity <span class="nt">--text-info</span></code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*</code></dd><dt>Appears if selection contains</dt><dd>Directories</dd><dt>Requirements</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">sudo </span>apt-get <span class="nb">install </span>zenity</code></pre></div>
<p>This means that you can use just filenames without a path in your custom actions to refer to a file in the current folder. This means that you can use the <code>%N</code> variable to process a list of selected files, instead if having to use the <code>%F</code> variable which includes the full pathname - this is handy for renaming just the files, without tampering with the pathname, for example.</p></section></section>
<section class="doc-section level-2"><h3 id="_variables">Variables</h3><p>Thunar custom actions can contain variable parameters, that get substituted with the actual value when you run the action. These allow you to refer to the files that are currently selected in Thunar when running your actions, without knowing in advance which ones. The following variables are available:</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 15%;"/><col style="width: 85%;"/></colgroup><thead><tr><th class="halign-left valign-top">This…</th><th class="halign-left valign-top">is replaced at runtime with this…</th></tr></thead><tbody><tr><td class="halign-left valign-top">%f</td><td class="halign-left valign-top">The path to the first selected file</td></tr><tr><td class="halign-left valign-top">%F</td><td class="halign-left valign-top">The paths to all the selected files</td></tr><tr><td class="halign-left valign-top">%d</td><td class="halign-left valign-top">Directory containing the file referred to by %f</td></tr><tr><td class="halign-left valign-top">%D</td><td class="halign-left valign-top">Directories containing the files referred to by %F</td></tr><tr><td class="halign-left valign-top">%n</td><td class="halign-left valign-top">The first selected filename, without the path</td></tr><tr><td class="halign-left valign-top">%N</td><td class="halign-left valign-top">All the selected filenames, without paths</td></tr></tbody></table></div>
<p>You can see examples of these variables in use below.</p></section></section>
<section class="doc-section level-1"><h2 id="_my_thunar_custom_actions">My Thunar Custom Actions</h2><p>To use these yourself, copy and paste these names, descriptions <span class="amp">&</span> commands into new actions in your Custom Actions Manager:</p>
<section class="doc-section level-2"><h3 id="_share_folder">Share Folder</h3><div class="dlist"><dl><dt>Description</dt><dd>Shares the currently selected folder, giving everyone read access.</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash">net usershare add %n %f <span class="s2">""</span> Everyone:R <span class="nv">guest_ok</span><span class="o">=</span>y</code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*</code></dd><dt>Appears if selection contains</dt><dd>Directories</dd></dl></div></section>
<section class="doc-section level-2"><h3 id="_flatten_folder">Flatten Folder</h3><div class="dlist"><dl><dt>Description</dt><dd>Moves all files from sub-folders to parent (current) folder, then removes all empty folders inside the current folder.</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash">find <span class="nb">.</span> <span class="nt">-mindepth</span> 2 <span class="nt">-type</span> f <span class="nt">-exec</span> <span class="nb">mv</span> <span class="s2">"{}"</span> <span class="nb">.</span> <span class="se">\;</span> <span class="o">&&</span> find <span class="nb">.</span> <span class="nt">-type</span> d <span class="nt">-empty</span> <span class="nt">-delete</span></code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*</code></dd><dt>Appears if selection contains</dt><dd>Directories</dd></dl></div></section>
<section class="doc-section level-2"><h3 id="_rename_to_lower_case">Rename to lower-case</h3><div class="dlist"><dl><dt>Description</dt><dd>Rename the currently selected files, making the filenames lower-case.</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="k">for </span>file <span class="k">in</span> %N<span class="p">;</span> <span class="k">do </span><span class="nb">mv</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> <span class="s2">"</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> | <span class="nb">tr</span> <span class="s1">'[:upper:]'</span> <span class="s1">'[:lower:]'</span><span class="si">)</span><span class="s2">"</span><span class="p">;</span> <span class="k">done</span></code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*</code></dd><dt>Appears if selection contains</dt><dd><em>All</em></dd></dl></div></section>
<section class="doc-section level-2"><h3 id="_slugify_filename">Slugify Filename</h3><div class="dlist"><dl><dt>Description</dt><dd>Rename the currently selected files, making the filenames lower-case <span class="amp">&</span> replacing spaces with dashes.</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="k">for </span>file <span class="k">in</span> %N<span class="p">;</span> <span class="k">do </span><span class="nb">mv</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> <span class="s2">"</span><span class="si">$(</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span> | <span class="nb">tr</span> <span class="nt">-s</span> <span class="s1">' '</span> | <span class="nb">tr</span> <span class="s1">' A-Z'</span> <span class="s1">'-a-z'</span> | <span class="nb">tr</span> <span class="nt">-s</span> <span class="s1">'-'</span> | <span class="nb">tr</span> <span class="nt">-c</span> <span class="s1">'[:alnum:][:cntrl:].'</span> <span class="s1">'-'</span><span class="si">)</span><span class="s2">"</span><span class="p">;</span> <span class="k">done</span></code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*</code></dd><dt>Appears if selection contains</dt><dd><em>All</em></dd></dl></div></section>
<section class="doc-section level-2"><h3 id="_copy_contents_to_clipboard">Copy Contents to Clipboard</h3><div class="dlist"><dl><dt>Description</dt><dd>Copies the contents of the selected file to the clipboard.</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">cat</span> <span class="s2">"%F"</span> | xclip <span class="nt">-i</span> <span class="nt">-selection</span> clipboard</code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*</code></dd><dt>Appears if selection contains</dt><dd>Text Files</dd><dt>Requirements</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">sudo </span>apt-get <span class="nb">install </span>xclip</code></pre></div></section>
<section class="doc-section level-2"><h3 id="_compare">Compare</h3><div class="dlist"><dl><dt>Description</dt><dd>Compares selected files or folders in <a href="http://meldmerge.org/">Meld</a></dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash">meld %F</code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*</code></dd><dt>Appears if selection contains</dt><dd>Directories, Text Files</dd><dt>Requirements</dt><dd>Either <a href="https://coderwall.com/p/isntfq">get the latest version of meld like this</a>, or install the version in your distributions repository:</dd></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">sudo </span>apt-get <span class="nb">install </span>meld</code></pre></div></section>
<section class="doc-section level-2"><h3 id="_compress_with_advpng">Compress with Advpng</h3><div class="dlist"><dl><dt>Description</dt><dd>Runs <a href="http://en.wikipedia.org/wiki/Advpng">Advpng</a> on each of the selected <span class="caps">PNG</span> Files.</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="k">for </span>file <span class="k">in</span> %F<span class="p">;</span> <span class="k">do </span>advpng <span class="nt">-z</span> <span class="nt">-4</span> <span class="nt">-q</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span><span class="p">;</span> <span class="k">done</span></code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*.png</code></dd><dt>Appears if selection contains</dt><dd>Image Files</dd><dt>Requirements</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">sudo </span>apt-get <span class="nb">install </span>advancecomp</code></pre></div></section>
<section class="doc-section level-2"><h3 id="_quantize_with_pngnq">Quantize with pngnq</h3><div class="dlist"><dl><dt>Description</dt><dd>Reduce to 8bit colour, by running <a href="https://github.com/stuart/pngnq">pngnq</a> on each of the selected <span class="caps">PNG</span> Files.</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="k">for </span>file <span class="k">in</span> %F<span class="p">;</span> <span class="k">do </span>pngnq <span class="nt">-s1</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span><span class="p">;</span> <span class="k">done</span></code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code>*.png</code></dd><dt>Appears if selection contains</dt><dd>Image Files</dd><dt>Requirements</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">sudo </span>apt-get <span class="nb">install </span>pngnq</code></pre></div></section>
<section class="doc-section level-2"><h3 id="_optimize_with_jpegoptim">Optimize with jpegoptim</h3><div class="dlist"><dl><dt>Description</dt><dd>Losslessly optimize JPEGs, by optimizing the Huffman tables and stripping comments and <span class="caps">EXIF</span> metadata from the file.</dd><dt>Command</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="k">for </span>file <span class="k">in</span> %F<span class="p">;</span> <span class="k">do </span>jpegoptim <span class="nt">--strip-all</span> <span class="nt">-of</span> <span class="s2">"</span><span class="nv">$file</span><span class="s2">"</span><span class="p">;</span> <span class="k">done</span></code></pre></div>
<div class="dlist"><dl><dt>File Pattern</dt><dd><code><strong>.jpg;</strong>.jpeg</code></dd><dt>Appears if selection contains</dt><dd>Image Files</dd><dt>Requirements</dt></dl></div>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="nb">sudo </span>apt-get <span class="nb">install </span>jpegoptim</code></pre></div></section></section>Post Statistics Plugin for Pelican2013-06-23T21:28:11-07:002013-06-23T21:28:11-07:00Duncan Locktag:duncanlock.net,2013-06-23:/blog/2013/06/23/post-statistics-plugin-for-pelican/<section id="preamble" aria-label="Preamble"><p>This is a Pelican plugin to calculate various statistics about a post and store them in an <code>article.stats</code> dictionary. You can see this in action in the sidebar on the left of this site.</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/post-statistics-plugin-for-pelican/pelican-plugin-post-stats-medium-example.png" alt="Nice touch from medium.com - now available in Pelican."></div>
<p>I wanted to implement the nice little “X min read” thing from <a href="https://medium.com/">Medium</a> - and it turned out that it was easy to provide a few other interesting stats at the same time, for people to use in their templates.</p>
<p>The returned <code>article.stats</code> dictionary contains the following:</p>
<div class="ulist"><ul><li><code>wc</code>: how many words</li><li><code>read_mins</code>: how many minutes would it take to read this article, <a href="http://en.wikipedia.org/wiki/Words_per_minute#Reading_and_comprehension">based on 250 wpm</a></li><li><code>word_counts</code>: frequency count of all the words in the article; can be used for tag/word clouds</li><li><code>fi</code>: <a href="http://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests">Flesch-kincaid</a> Index/ Reading Ease</li><li><code>fk</code>: Flesch-kincaid Grade Level</li></ul></div>
<p>For example:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="p">{</span>
<span class="s">'wc'</span><span class="p">:</span> <span class="mi">2760</span><span class="p">,</span>
<span class="s">'fi'</span><span class="p">:</span> <span class="s">'65.94'</span><span class="p">,</span>
<span class="s">'fk'</span><span class="p">:</span> <span class="s">'7.65'</span><span class="p">,</span>
<span class="s">'word …</span></code></pre></div></section><section id="preamble" aria-label="Preamble"><p>This is a Pelican plugin to calculate various statistics about a post and store them in an <code>article.stats</code> dictionary. You can see this in action in the sidebar on the left of this site.</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/post-statistics-plugin-for-pelican/pelican-plugin-post-stats-medium-example.png" alt="Nice touch from medium.com - now available in Pelican."></div>
<p>I wanted to implement the nice little “X min read” thing from <a href="https://medium.com/">Medium</a> - and it turned out that it was easy to provide a few other interesting stats at the same time, for people to use in their templates.</p>
<p>The returned <code>article.stats</code> dictionary contains the following:</p>
<div class="ulist"><ul><li><code>wc</code>: how many words</li><li><code>read_mins</code>: how many minutes would it take to read this article, <a href="http://en.wikipedia.org/wiki/Words_per_minute#Reading_and_comprehension">based on 250 wpm</a></li><li><code>word_counts</code>: frequency count of all the words in the article; can be used for tag/word clouds</li><li><code>fi</code>: <a href="http://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests">Flesch-kincaid</a> Index/ Reading Ease</li><li><code>fk</code>: Flesch-kincaid Grade Level</li></ul></div>
<p>For example:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="p">{</span>
<span class="s">'wc'</span><span class="p">:</span> <span class="mi">2760</span><span class="p">,</span>
<span class="s">'fi'</span><span class="p">:</span> <span class="s">'65.94'</span><span class="p">,</span>
<span class="s">'fk'</span><span class="p">:</span> <span class="s">'7.65'</span><span class="p">,</span>
<span class="s">'word_counts'</span><span class="p">:</span> <span class="n">Counter</span><span class="p">({</span><span class="sa">u</span><span class="s">'to'</span><span class="p">:</span> <span class="mi">98</span><span class="p">,</span> <span class="sa">u</span><span class="s">'a'</span><span class="p">:</span> <span class="mi">90</span><span class="p">,</span> <span class="p">...}),</span>
<span class="s">'read_mins'</span><span class="p">:</span> <span class="mi">12</span>
<span class="p">}</span></code></pre></div>
<p>This allows you to output these values in your templates, like this, for example:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html+jinja"><p title="~{{ article.stats['wc'] }} words">~{{ article.stats['read_mins'] }} min read</p>
<ul>
<li>Flesch-kincaid Index/ Reading Ease: {{ article.stats['fi'] }}</li>
<li>Flesch-kincaid Grade Level: {{ article.stats['fk'] }}</li>
</ul></code></pre></div>
<p>The <code>word_counts</code> variable is a python <code>Counter</code> dictionary and looks something like this, with each unique word and it’s frequency:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">Counter</span><span class="p">({</span><span class="sa">u</span><span class="s">'to'</span><span class="p">:</span> <span class="mi">98</span><span class="p">,</span> <span class="sa">u</span><span class="s">'a'</span><span class="p">:</span> <span class="mi">90</span><span class="p">,</span> <span class="sa">u</span><span class="s">'the'</span><span class="p">:</span> <span class="mi">83</span><span class="p">,</span> <span class="sa">u</span><span class="s">'of'</span><span class="p">:</span> <span class="mi">50</span><span class="p">,</span> <span class="sa">u</span><span class="s">'karma'</span><span class="p">:</span> <span class="mi">50</span><span class="p">,</span> <span class="p">.....</span></code></pre></div>
<p>This can be used to create a tag/word cloud for a post.</p></section>
<section class="doc-section level-1"><h2 id="_requirements">Requirements</h2><p><code>post_stats</code> requires <a href="http://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup</a>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pip <span class="nb">install </span>beautifulsoup4</code></pre></div></section>
<section class="doc-section level-1"><h2 id="_caveat_emptor">Caveat Emptor</h2><p>Please note that the values are a wee bit approximate - it’s surprisingly difficult to compute a <em>perfect</em> word count - it depends on what you count as a ‘word’, how many code samples, data tables, etc…​ you have in your post. Calculating the <a href="http://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests">Flesch-kincaid Index/ Reading Ease/ Grade level</a> stuff involves counting syllables - again something that is very hard to do perfectly, but quite easy to get 90% right.</p>
<p>In addition, the Flesch-kincaid stuff currently only works on English text, but everything else should work multi-language. I haven’t done much Unicode testing though, so <a href="https://github.com/getpelican/pelican-plugins">patches welcome</a>!</p></section>A Marvellous & Incomplete Compendium of reddit Automatons & Bots2013-06-19T19:48:54-07:002021-06-12T08:52:41-07:00Duncan Locktag:duncanlock.net,2013-06-19:/blog/2013/06/19/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/<figure class="image-block"><a class="image" href="http://openclipart.org/detail/1654/robot-by-johnny_automatic"><img alt="reddit bots diagram" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-diagram.png"/></a>
<figcaption>Figure 1. How much does a software bot weigh, anyway? Heavily modified <span class="amp">&</span> adapted from the original public domain robot on openclipart, posted by johnny_automatic.</figcaption></figure>
<p>reddit, the insanely popular internet community, had 71,435,935 unique visitors last month, with over 2,360,783 people logged in <a class="footnote-ref" href="#_footnote_1" id="_footnoteref_1" role="doc-noteref" title="View footnote 1">[1]</a>.</p>
<p>I say people - but it turns out that not all the denizens of reddit are human. There are also bots. Lots and lots of bots. How many? No-one really knows. <a class="footnote-ref" href="#_footnote_2" id="_footnoteref_2" role="doc-noteref" title="View footnote 2">[2]</a></p>
<p>This is an interesting and somewhat shadowy facet of the otherwise very public reddit community, so I thought I’d take a closer look…</p>
<section class="doc-section level-1"><h2 id="_what_is_a_reddit_bot">What <em>is</em> a reddit bot?</h2><p>A reddit bot is no different from any other user, as far as reddit is concerned. The only difference is that rather than a human logging in to upvote cat pictures and …</p></section><figure class="image-block"><a class="image" href="http://openclipart.org/detail/1654/robot-by-johnny_automatic"><img alt="reddit bots diagram" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-diagram.png"/></a>
<figcaption>Figure 1. How much does a software bot weigh, anyway? Heavily modified <span class="amp">&</span> adapted from the original public domain robot on openclipart, posted by johnny_automatic.</figcaption></figure>
<p>reddit, the insanely popular internet community, had 71,435,935 unique visitors last month, with over 2,360,783 people logged in <a class="footnote-ref" href="#_footnote_1" id="_footnoteref_1" role="doc-noteref" title="View footnote 1">[1]</a>.</p>
<p>I say people - but it turns out that not all the denizens of reddit are human. There are also bots. Lots and lots of bots. How many? No-one really knows. <a class="footnote-ref" href="#_footnote_2" id="_footnoteref_2" role="doc-noteref" title="View footnote 2">[2]</a></p>
<p>This is an interesting and somewhat shadowy facet of the otherwise very public reddit community, so I thought I’d take a closer look…</p>
<section class="doc-section level-1"><h2 id="_what_is_a_reddit_bot">What <em>is</em> a reddit bot?</h2><p>A reddit bot is no different from any other user, as far as reddit is concerned. The only difference is that rather than a human logging in to upvote cat pictures and post comments, this account is used by an automated computer script.</p>
<p>reddit has a captcha system to help prevent automated signups - so the bot’s human creator will need to create a new user account for the bot to use, then program the username and password into the bot, so it can use that account - just like a human would.</p>
<p>In the same way that computers can run scripts to automatically check weather data and <a href="https://ifttt.com/search/query/umbrella">send you a message if you should take an umbrella with you today</a>, a computer can run scripts to automatically check reddit for certain activity - and post comments if certain conditions are met. <a class="footnote-ref" href="#_footnote_3" id="_footnoteref_3" role="doc-noteref" title="View footnote 3">[3]</a> A reddit bot does this by doing what your web browser does behind the scenes when you use websites - it makes requests and sends things to the reddit servers.</p>
<p>reddit also has an <span class="caps">API</span> (Application Programming Interface) <a class="footnote-ref" href="#_footnote_4" id="_footnoteref_4" role="doc-noteref" title="View footnote 4">[4]</a> which makes it easier for automated external services (including bots) to talk to reddit. This allows bots <span class="amp">&</span> scrapers to do things in bulk, like “Show me all the new posts in the last hour” – they can then go away, analyze this information and then take actions based on it.</p></section>
<section class="doc-section level-1"><h2 id="_its_a_hard_bot_life">It’s a Hard Bot Life</h2><p>A major problem for any internet activity at scale is ‘spam’, or similar unwanted activity, in some form or other.</p>
<p>reddit has scale coming out of its ears, so needs aggressive, pervasive and rapid automated spam control algorithms - combined with extensive human flagging and moderation - just to survive.</p>
<p>These mechanisms come down on bots particularly hard, to prevent the place being overrun by an army of implacable text hurling machines, typing <span class="amp">&</span> posting at the speed of light.</p>
<p>This is a <a href="http://www.reddit.com/r/InternetAMA/comments/1gescq/i_am_tictactoebot_i_derail_threads_and_i_am/">really nice /r/InternetAmA thread discussing the retirement of TicTacToeBot</a>, which did exactly what you’d expect:</p>
<figure class="image-block"><img alt="reddit bots tictactoebot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-tictactoebot-example.png"/>
<figcaption>Figure 2. The only winning move is not to play.</figcaption></figure>
<p>Some quotes from TicTacToeBots developer:</p>
<div class="quote-block"><blockquote><p>Q: Your account has only existed for 6 days. How have you already been banned from the subreddits?</p>
<p>A: No clue. Every day I am banned from subreddits. I woke up today, looked at the mail. Banned from circlejerk, cats, and skyrim… Those aren’t even default subs. I had to sub to new channels to be any active as a bot.</p>
<p>Q: You actually got banned from circlejerk <a class="footnote-ref" href="#_footnote_5" id="_footnoteref_5" role="doc-noteref" title="View footnote 5">[5]</a>? For derailing threads?!?</p>
<p>A: Yes</p></blockquote></div>
<p>TicTacToeBot got shot down for derailing threads - i.e. for being disruptive. It was <a href="http://www.reddit.com/r/todayilearned/comments/1fzgle/til_that_110_people_once_tied_for_second_prize_in/cafg3xj?context=2">intentionally designed to randomly pop up and challenge people</a> to a game - whimsical and fun, but also an uninvited disruption - albeit a harmless good-natured one.</p>
<p>So unless they’re careful, disruptive bots tend to have a fairly short life on reddit, quickly being hunted down and blocked by reddit’s immune system – even whimsical and seemingly harmless bots, like TicTacToeBot.</p>
<p>More circumspect bots, like <a href="#_jiffybot">JiffyBot</a>, <a href="#_chart_bot">CHART_BOT</a> or <a href="#_serendipitybot_rserendipity">serendipity</a> - either completely or largely confine themselves to their own subreddits, only turning up elsewhere when invited. This generally means that they’ll be left alone to do their thing, because they’re not interfering with anyone else. They’re also completely upfront about what they do and provide a useful service to the reddit community.</p>
<section class="doc-section level-2"><h3 id="_workin_on_a_bot_farm">Workin’ on a Bot Farm</h3><p>Bots also take resources to run - both to initially create <span class="amp">&</span> then to maintain the code - but mainly to provide a computer to run them on <a class="footnote-ref" href="#_footnote_6" id="_footnoteref_6" role="doc-noteref" title="View footnote 6">[6]</a>. Bots need a computer to host their code and to lavish <span class="caps">CPU</span> cycles running them - reddit doesn’t do this, it’s up to the bots creator to host them somewhere. This generally isn’t free and can eat up quite a lot of computer resources, depending on what the bot does. Bots can get shut down by their creators for lack of resources - time or money - or lack of interest. Pretty much all reddit bots are just created for fun, for learning, or both - sometimes the creator just wants to move on to another project.</p></section>
<section class="doc-section level-2"><h3 id="_bad_bots_sad_bots">Bad Bots, Sad Bots</h3><p>Some bots are designed to try to behave statistically more like human users <a class="footnote-ref" href="#_footnote_7" id="_footnoteref_7" role="doc-noteref" title="View footnote 7">[7]</a>, or to deliberately try to slip under the radar. Some bots are designed to boost the reddit karma <a class="footnote-ref" href="#_footnote_8" id="_footnoteref_8" role="doc-noteref" title="View footnote 8">[8]</a> of their masters by pretending to be regular users and up-voting their masters posts and down-voting those who disagree with them. Some bots are designed to start flame wars and generally be mean, virtually.</p>
<p>This is pretty sad and pathetic… so I’m going to ignore them.</p>
<p>So, without further ado, here’s the compendium, split into <a href="#_bots_that_you_can_summon_with_an_incantation">Bots that you can Summon with an Incantation</a> and <a href="#_bots_that_just_show_up_without_human_intervention">Bots that just Show Up, without human intervention</a>.</p></section></section>
<section class="doc-section level-1"><h2 id="_bots_that_you_can_summon_with_an_incantation">Bots that you can Summon with an Incantation</h2><p>These bots listen out for their summoning incantation to be posted somewhere on reddit, then turn up and do their thing in response:</p>
<section class="doc-section level-2"><h3 id="_jiffybot">JiffyBot</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Makes animated GIFs out of YouTube links</dd><dt>Creators</dt><dd><ul><li><a href="http://www.reddit.com/user/DrKabob">/u/DrKabob</a></li><li><a href="http://www.reddit.com/user/GoogaNautGod">/u/GoogaNautGod</a></li></ul></dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/JiffyBot">/u/JiffyBot</a></li><li><a href="http://www.reddit.com/r/JiffyBot">/r/JiffyBot</a></li><li><a href="http://www.reddit.com/r/JiffyBot/comments/1fp9qh/how_do_i_summon_jiffy_bot/">JiffyBot Documentation</a></li><li><a href="http://www.reddit.com/r/JiffyBot/comments/1fvrsq/the_official_make_your_own_gif_verison_sfw/">JiffyBot in Action</a></li><li><a href="http://www.reddit.com/r/JiffyBot/comments/1fwo0y/jiffy_bot_feedback_and_questions_faq/">JiffyBot <span class="caps">FAQ</span></a></li><li><a href="https://github.com/l1am9111/JiffyBot">JiffyBot Source Code</a> - <span class="caps">NB</span> this is an orphaned fork of the original GitHub code repository; I’m currently trying to find out what happened to the original.</li></ul></dd><dt>Current Karma</dt><dd><ul><li>1 link karma</li><li>30,173 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>16 days</dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 33.6633%;"/><col style="width: 35.6435%;"/><col style="width: 30.6932%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/JiffyBot</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">333 (391)</td></tr><tr><td class="halign-left valign-top">/r/cringe</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">92 (614)</td></tr><tr><td class="halign-left valign-top">/r/tf2</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">45 (315)</td></tr><tr><td class="halign-left valign-top">/r/gaming</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">40 (418)</td></tr><tr><td class="halign-left valign-top">/r/youtubehaiku</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">36 (173)</td></tr><tr><td class="halign-left valign-top">/r/leagueoflegends</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">27 (73)</td></tr><tr><td class="halign-left valign-top">/r/funny</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">27 (434)</td></tr><tr><td class="halign-left valign-top">/r/YouShouldKnow</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">27 (28)</td></tr><tr><td class="halign-left valign-top">/r/SeeThisShit</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">22 (22)</td></tr><tr><td class="halign-left valign-top">/r/DotA2</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">17 (35)</td></tr><tr><td class="halign-left valign-top">/r/starcraft</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">15 (96)</td></tr><tr><td class="halign-left valign-top">/r/hockey</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">12 (7)</td></tr><tr><td class="halign-left valign-top">/r/atheism</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">10 (221)</td></tr></tbody></table></div>
<p>Summon by posting a link to a YouTube video, then writing <code>Jiffy!</code> followed by a start time and end time, in either of these forms:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="n">Jiffy</span><span class="err">!</span> <span class="mi">0</span><span class="p">:</span><span class="mi">07</span><span class="o">-</span><span class="mi">0</span><span class="p">:</span><span class="mi">12</span>
<span class="o">/</span><span class="n">u</span><span class="o">/</span><span class="n">JiffyBot</span> <span class="mi">0</span><span class="p">:</span><span class="mi">00</span><span class="o">-</span><span class="mi">0</span><span class="p">:</span><span class="mi">15</span></code></pre></div>
<p>The second form is apparently more reliable.</p>
<p>The bot will respond by replying to your comment, with a comment of it’s own, containing an <a href="http://imgur.com/">imgur.com</a> link to an animated <span class="caps">GIF</span> of that video, for the time period you specified. This is great for people on mobile devices - animated GIFs load <em>much</em> quicker than YouTube.</p>
<figure class="image-block"><img alt="reddit bots jiffybot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-jiffybot-example.png"/>
<figcaption>Figure 3. JiffyBot in action: it can also do multiple GIFs!</figcaption></figure></section>
<section class="doc-section level-2"><h3 id="_bitcointip">BitcoinTip</h3><div class="dlist"><dl><dt>Purpose</dt><dd>The bitcointip bot allows redditors to tip each other ‘real’ money, just by leaving a reddit comment or message.</dd><dt>Human Creator</dt><dd><ul><li><a href="http://www.reddit.com/user/NerdfighterSean">/u/NerdfighterSean</a></li></ul></dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/bitcointip">/u/bitcointip</a></li><li><a href="http://www.reddit.com/r/bitcointip">/r/bitcointip</a></li><li><a href="http://www.reddit.com/r/bitcointip/comments/13iykn/_bitcointipdocumentation/">BitcoinTip Documentation</a></li><li><a href="http://imgur.com/CwDYZqW">BitcoinTip Quickstart Guide</a></li><li><a href="https://github.com/NerdfighterSean/bitcointip">Source Code</a> - rather out of date.</li></ul></dd><dt>Current Karma</dt><dd><ul><li>9 link karma</li><li>11,906 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>1 year</dd><dt>Source Code</dt><dd><a href="https://github.com/NerdfighterSean/bitcointip">https://github.com/NerdfighterSean/bitcointip</a></dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 35%;"/><col style="width: 35%;"/><col style="width: 30%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/Bitcoin</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">368 (813)</td></tr><tr><td class="halign-left valign-top">/r/GirlsGoneBitcoin</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">51 (59)</td></tr><tr><td class="halign-left valign-top">/r/worldnews</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">36 (133)</td></tr><tr><td class="halign-left valign-top">/r/IAmA</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">30 (81)</td></tr><tr><td class="halign-left valign-top">/r/AskReddit</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">30 (88)</td></tr><tr><td class="halign-left valign-top">/r/bitcointip</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">29 (49)</td></tr><tr><td class="halign-left valign-top">/r/pics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">20 (136)</td></tr><tr><td class="halign-left valign-top">/r/technology</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">13 (134)</td></tr><tr><td class="halign-left valign-top">/r/AdviceAnimals</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">12 (23)</td></tr><tr><td class="halign-left valign-top">/r/investing</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">11 (43)</td></tr><tr><td class="halign-left valign-top">/r/gaming</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">11 (241)</td></tr><tr><td class="halign-left valign-top">/r/tf2</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">10 (145)</td></tr><tr><td class="halign-left valign-top">/r/starcraft</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">10 (205)</td></tr></tbody></table></div>
<p>The bot scans user comments and messages for tips of the form:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">+/u/bitcointip @RedditUsername $1
+/u/bitcointip @Username $1usd
+/u/bitcointip BitcoinAddress 1 millibit
+/u/bitcointip Username ฿0.001 verify
+/u/bitcointip $1 # This tips 1 usd to whoever posted the comments parent
+/u/bitcointip BitcoinAddress ALL # This sends your entire balance to that bitcoin address
+/u/bitcointip 2 internets # An "internet" is worth $0.25</code></pre></div>
<p>You have to setup a bitcointip tip account in advance and put some funds into it. It then sends the specified amount of bitcoins from the sender’s bitcointip account, to the receiver’s bitcointip account. Supports lots of different currencies, which get converted to bitcoin automatically.</p>
<p>Allows you to tip people for useful or awesome comments, in a very natural and low friction way:</p>
<figure class="image-block"><a class="image" href="http://en.wikipedia.org/wiki/Adam_Savage"><img alt="reddit bots bitcointip example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-bitcointip-example.png"/></a>
<figcaption>Figure 4. BitcoinTip in action: Adam Savage gets tipped. Yes that Adam Savage.</figcaption></figure></section>
<section class="doc-section level-2"><h3 id="_chart_bot">CHART_BOT</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Automatically generates and posts a chart of your posting history - or someone else’s.</dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/CHART_BOT">/u/CHART_BOT</a></li><li><a href="http://www.reddit.com/r/CHART_BOT">/r/CHART_BOT</a></li></ul></dd><dt>Active SubReddits</dt><dd>Overwhelmingly active in it’s own subreddit, but has been known to pop-up elsewhere, for the lulz:</dd></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 45%;"/><col style="width: 30%;"/><col style="width: 25%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/CHART_BOT</td><td class="halign-left valign-top">1 (2)</td><td class="halign-left valign-top">931 (1063)</td></tr><tr><td class="halign-left valign-top">/r/<span class="caps">WTF</span></td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">19 (13)</td></tr><tr><td class="halign-left valign-top">/r/wheredidthesodago</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">14 (-14)</td></tr><tr><td class="halign-left valign-top">/r/science</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">13 (13)</td></tr><tr><td class="halign-left valign-top">/r/TheLastAirbender</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">12 (20)</td></tr></tbody></table></div>
<div class="dlist"><dl><dt>Current Karma</dt><dd><ul><li>3 link karma</li><li>5,686 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>8 months</dd></dl></div>
<p>Making a submission <a href="http://www.reddit.com/r/CHART_BOT">to this subreddit</a> will cause CHART_BOT to automatically generate and post a chart of your reddit posting history. You can also request charts of other reddit users by putting their username prefixed with an @ in the title of your submission. The charts look like this - <a href="http://www.reddit.com/r/CHART_BOT/comments/1gdpu9/chart_me_up_baby/">here’s mine</a>:</p>
<div class="image-block"><img alt="Screenshot of CHART_BOTS output for duncanlock, as of June 2013." src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/duncan-locks-chart-bot-chart-june-2013.png"/></div>
<p>CHART_BOT also produces some graphs of activity which are quite interesting. Here are the ‘Posts Over Time’ ones for me (on the left) and chartbot (on the right). You can clearly see the characteristic posting pattern of humans (irregular) vs. bots (regular):</p>
<figure class="image-block"><img alt="Two scatter plots of reddit postings, over time. Left one for human user duncanlock, right one for chart_bot." src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-duncanlock-chartbot-postings-over-time-graph.png"/>
<figcaption>Figure 5. Fairly typical human reddit user (left) vs bot (right). Bot scripts are often run on a regular schedule - e.g. once an hour, every 10 minutes, etc… - which explains the regular patterns of activity.</figcaption></figure></section></section>
<section class="doc-section level-1"><h2 id="_bots_that_just_show_up_without_human_intervention">Bots that just Show Up, without human intervention</h2><p>These bots ceaselessly scan the endless, mighty cataract of text that is reddit and leap in whenever they sense patterns in the noise <span class="amp">&</span> spume that match their programming.</p>
<section class="doc-section level-2"><h3 id="_raddit_bot">raddit-bot</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Shares (most of) the data about the posts it sees being used on <a href="http://radd.it/">radd.it</a>. Currently it’s sharing a combination of data from youtube, soundcloud, vimeo, last.fm, IMDb, and amazon; only comments in subreddits it’s been invited to.</dd><dt>Human Creator</dt><dd><ul><li><a href="http://www.reddit.com/user/radd_it">/u/radd_it</a></li></ul></dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/raddit-bot">/u/raddit-bot</a></li><li><a href="http://www.reddit.com/r/radd_it">/r/raddit-bot</a></li><li><a href="http://www.reddit.com/r/radd_it/comments/1gxa85/who_is_uradditbot_and_why_is_it_commenting_here/">raddit-bot <span class="caps">FAQ</span></a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>1915 link karma</li><li>376 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>1 month</dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 35%;"/><col style="width: 35%;"/><col style="width: 30%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/listentothis</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">765 (1109)</td></tr><tr><td class="halign-left valign-top">/r/FullMoviesOnline</td><td class="halign-left valign-top">352 (764)</td><td class="halign-left valign-top">213 (215)</td></tr><tr><td class="halign-left valign-top">/r/listentonew</td><td class="halign-left valign-top">51 (55)</td><td class="halign-left valign-top">0</td></tr><tr><td class="halign-left valign-top">/r/<span class="caps">VBT</span></td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">1 (1)</td></tr><tr><td class="halign-left valign-top">/r/Music</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">1 (2)</td></tr></tbody></table></div>
<p>Raddit-bot is a helpful bot that posts information when you post a link to a piece of media that’s been on <a href="http://radd.it/">radd.it</a>. It’s posts look like this, sharing a wealth of links and information about things that people have linked to:</p>
<div class="image-block"><img alt="reddit bots radditbot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-radditbot-example.png"/></div>
<p>Discovered this bot while browsing <a href="http://www.reddit.com/r/listentothis">/r/listentothis</a> - which in turn led me to discover <a href="http://radd.it/">radd.it</a>; I’m currently trying to resist getting distracted by radd.it itself.</p></section>
<section class="doc-section level-2"><h3 id="_haiku_robot">haiku_robot</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Watches reddit for comments that would qualify as Haiku <a class="footnote-ref" href="#_footnote_9" id="_footnoteref_9" role="doc-noteref" title="View footnote 9">[9]</a> and posts a reply, with the original text reformatted into 3 lines of 5, 7 <span class="amp">&</span> 5 syllables.</dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/u/haiku_robot">/u/haiku_robot</a></li><li><a href="http://www.reddit.com/r/IAmA/comments/1fr7c5/beep_boop_beep_boop_bopiama_haiku_robotask_me/">haiku_robot <span class="caps">FAQ</span></a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>1 link karma</li><li>104,473 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>1 year</dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 36%;"/><col style="width: 34%;"/><col style="width: 30%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/funny</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">284 (2580)</td></tr><tr><td class="halign-left valign-top">/r/pics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">199 (1239)</td></tr><tr><td class="halign-left valign-top">/r/AdviceAnimals</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">126 (619)</td></tr><tr><td class="halign-left valign-top">/r/gaming</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">90 (501)</td></tr><tr><td class="halign-left valign-top">/r/<span class="caps">WTF</span></td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">56 (618)</td></tr><tr><td class="halign-left valign-top">/r/todayilearned</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">25 (115)</td></tr><tr><td class="halign-left valign-top">/r/IAmA</td><td class="halign-left valign-top">1 (8)</td><td class="halign-left valign-top">23 (99)</td></tr><tr><td class="halign-left valign-top">/r/gifs</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">23 (164)</td></tr><tr><td class="halign-left valign-top">/r/videos</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">22 (77)</td></tr><tr><td class="halign-left valign-top">/r/leagueoflegends</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">15 (404)</td></tr><tr><td class="halign-left valign-top">/r/mildlyinteresting</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">15 (28)</td></tr><tr><td class="halign-left valign-top">/r/gonewild</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">10 (116)</td></tr><tr><td class="halign-left valign-top">/r/technology</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">10 (13)</td></tr><tr><td class="halign-left valign-top">Plus 45 more…</td><td class="halign-left valign-top"></td><td class="halign-left valign-top"></td></tr></tbody></table></div>
<p>This seems to be quite popular, with lots of very highly upvoted comments - like this one:</p>
<div class="image-block"><img alt="reddit bots haikubot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-haikubot-example.png"/></div></section>
<section class="doc-section level-2"><h3 id="_metric_system_converting_bot">Metric System Converting bot</h3><div class="dlist"><dl><dt>Purpose</dt><dd>When it sees a post using Imperial/<span class="caps">US</span> units, it replies with a conversion to their Metric equivalents.</dd><dt>Human Creator</dt><dd><ul><li><a href="http://www.reddit.com/user/xwcg">/u/xwcg</a></li></ul></dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/MetricConversionBot">/u/MetricConversionBot</a></li><li><a href="http://www.reddit.com/r/MetricConversionBot">/r/MetricConversionBot</a></li><li><a href="http://www.reddit.com/r/MetricConversionBot/comments/1f53fw/faq/">MetricConversionBot <span class="caps">FAQ</span></a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>239 link karma</li><li>26,779 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>27 days</dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 31.6831%;"/><col style="width: 36.6336%;"/><col style="width: 31.6833%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/AdviceAnimals</td><td class="halign-left valign-top">1 (285)</td><td class="halign-left valign-top">538 (4160)</td></tr><tr><td class="halign-left valign-top">/r/pics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">94 (1878)</td></tr><tr><td class="halign-left valign-top">/r/todayilearned</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">68 (625)</td></tr><tr><td class="halign-left valign-top">/r/gaming</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">63 (65)</td></tr><tr><td class="halign-left valign-top">/r/videos</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">44 (493)</td></tr><tr><td class="halign-left valign-top">/r/gifs</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">15 (258)</td></tr><tr><td class="halign-left valign-top">/r/politics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">15 (230)</td></tr><tr><td class="halign-left valign-top">/r/progresspics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">10 (92)</td></tr></tbody></table></div>
<p>MetricConversionBot will convert the following units to their metric equivalents:</p>
<div class="ulist"><ul><li>Pounds (lbs) to Kilograms</li><li>Miles to Kilometers</li><li>Miles per hour to Kilometers per Hour</li><li>Foot/Feet to Meters</li><li>Kelvin to Celsius</li><li>Fahrenheit to Celsius</li><li>inch to cm</li><li>yard to meters</li><li>(<span class="caps">US</span>) fl. oz. to ml</li><li>ounces to grams</li></ul></div>
<p>and it leaves comments that look like this:</p>
<div class="image-block"><img alt="reddit bots metricconversionbot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-metricconversionbot-example.png"/></div>
<p>This bot is a (<a href="http://www.reddit.com/r/TheoryOfReddit/comments/1fop0k/why_is_umetricmonversionmot_succeeding_while_usi/">more popular</a>) successor to the deceased <a href="http://www.reddit.com/user/si_bot">SI_BOT</a>. Interestingly, MetricConversionBot has attracted it’s own parody bots, <a href="http://www.reddit.com/user/MetricConversionNot">MetricConversionNot</a> - which randomly makes similar looking, but factually inaccurate parody comments (somewhat similar to the older, inactive parody bot <a href="http://www.reddit.com/user/Lord-Longbottom">Lord_Longbottom</a>) and <a href="http://www.reddit.com/user/UselessConversionBot">UselessConversionBot</a>:</p>
<div class="image-block"><img alt="reddit bots uselessconversionbot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-uselessconversionbot-example.png"/></div></section>
<section class="doc-section level-2"><h3 id="_website_mirror_bot">Website Mirror bot</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Mirrors websites that go down from the traffic surge, due to being posted on reddit.</dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/Website_Mirror_Bot">/u/Website_Mirror_Bot</a></li><li><a href="http://www.reddit.com/r/Website_Mirror_Bot">/r/Website_Mirror_Bot</a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>1 link karma</li><li>9,946 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>20 days</dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 33.6633%;"/><col style="width: 35.6435%;"/><col style="width: 30.6932%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/todayilearned</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">29 (6391)</td></tr><tr><td class="halign-left valign-top">/r/politics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">17 (870)</td></tr><tr><td class="halign-left valign-top">/r/worldnews</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">15 (1021)</td></tr><tr><td class="halign-left valign-top">/r/technology</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">8 (203)</td></tr><tr><td class="halign-left valign-top">/r/Bitcoin</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">4 (25)</td></tr><tr><td class="halign-left valign-top">/r/atheism</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">4 (2299)</td></tr><tr><td class="halign-left valign-top">/r/starcraft</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">4 (50)</td></tr><tr><td class="halign-left valign-top">/r/conspiracy</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">4 (15)</td></tr><tr><td class="halign-left valign-top">/r/leagueoflegends</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">3 (109)</td></tr></tbody></table></div>
<p>Takes a (generally very tall) <a href="http://i.imgur.com/MyiPyDE.jpg">screenshot</a> of the page that was linked to, puts it on imgur.com and posts a link in a comment:</p>
<div class="image-block"><img alt="reddit bots websitemirrorbot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-websitemirrorbot-example.png"/></div></section>
<section class="doc-section level-2"><h3 id="_tabledresser">tabledresser</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Automatically generates a summary table from an <a href="http://www.reddit.com/r/IAmA/">AmA thread</a>, showing all answered questions, along with their answers.</dd><dt>Human Creator</dt><dd><ul><li><a href="http://www.reddit.com/u/epsy">/u/epsy</a></li></ul></dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/tabledresser">/u/tabledresser</a></li><li><a href="http://www.reddit.com/r/tabled">/r/tabled</a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>4 link karma</li><li>8,857 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>1 year</dd><dt>Source Code</dt><dd><a href="https://github.com/epsy/tabledresser">https://github.com/epsy/tabledresser</a></dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 29%;"/><col style="width: 38%;"/><col style="width: 33%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/tabled</td><td class="halign-left valign-top">1000 (9253)</td><td class="halign-left valign-top">0</td></tr><tr><td class="halign-left valign-top">/r/IAmA</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">970 (4377)</td></tr><tr><td class="halign-left valign-top">/r/InternetAMA</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">19 (62)</td></tr><tr><td class="halign-left valign-top">/r/tf2trade</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">2 (4)</td></tr></tbody></table></div>
<p>It posts the first few rows in the actual AmA thread, with a link to the full table that it posts to <a href="http://www.reddit.com/r/tabled">/r/tabled</a>. This provides a great way to quickly read a condensed summary of a complete AmA thread, <a href="http://www.reddit.com/r/tabled/comments/1g9nja/table_iama_i_am_james_bamford_one_of_the/">like this one</a>. They look something like this:</p>
<div class="image-block"><img alt="reddit bots tabledresserbot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-tabledresserbot-example.png"/></div></section>
<section class="doc-section level-2"><h3 id="_videolinkbot">VideoLinkBot</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Posts a summary of all video links in a discussion, kept up to date as the discussion grows.</dd><dt>Human Creator</dt><dd><ul><li><a href="http://www.reddit.com/user/shaggorama">/u/shaggorama</a></li></ul></dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/VideoLinkBot">/u/VideoLinkBot</a></li><li><a href="http://www.reddit.com/r/VideoLinkBot/">/r/VideoLinkBot</a></li><li><a href="http://www.reddit.com/r/VideoLinkBot/wiki/faq">VideoLinkBot <span class="caps">FAQ</span></a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>25 link karma</li><li>49,423 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>4 months</dd><dt>Source Code</dt><dd><a href="https://github.com/dmarx/VideoLinkBot">https://github.com/dmarx/VideoLinkBot</a></dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 40%;"/><col style="width: 32%;"/><col style="width: 28%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/videos</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">126 (343)</td></tr><tr><td class="halign-left valign-top">/r/gaming</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">93 (167)</td></tr><tr><td class="halign-left valign-top">/r/hiphopheads</td><td class="halign-left valign-top">1 (0)</td><td class="halign-left valign-top">48 (123)</td></tr><tr><td class="halign-left valign-top">/r/leagueoflegends</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">47 (118)</td></tr><tr><td class="halign-left valign-top">/r/todayilearned</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">41 (69)</td></tr><tr><td class="halign-left valign-top">/r/movies</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">23 (66)</td></tr><tr><td class="halign-left valign-top">/r/nfl</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">21 (86)</td></tr><tr><td class="halign-left valign-top">/r/nba</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">18 (32)</td></tr><tr><td class="halign-left valign-top">/r/politics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">18 (19)</td></tr><tr><td class="halign-left valign-top">/r/Random_Acts_Of_Amazon</td><td class="halign-left valign-top">4 (98)</td><td class="halign-left valign-top">13 (21)</td></tr><tr><td class="halign-left valign-top">/r/WhereDoIStart</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">16 (36)</td></tr><tr><td class="halign-left valign-top">/r/hockey</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">15 (39)</td></tr><tr><td class="halign-left valign-top">/r/SquaredCircle</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">15 (43)</td></tr><tr><td class="halign-left valign-top">/r/worldnews</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">14 (27)</td></tr><tr><td class="halign-left valign-top">/r/IAmA</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">12 (263)</td></tr><tr><td class="halign-left valign-top">/r/<span class="caps">CFB</span></td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">12 (33)</td></tr><tr><td class="halign-left valign-top">/r/DotA2</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">12 (28)</td></tr><tr><td class="halign-left valign-top">/r/tipofmytongue</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">12 (14)</td></tr><tr><td class="halign-left valign-top">/r/teenagers</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">11 (21)</td></tr><tr><td class="halign-left valign-top">/r/VideoLinkBot</td><td class="halign-left valign-top">11 (17)</td><td class="halign-left valign-top">0</td></tr><tr><td class="halign-left valign-top">/r/atheism</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">10 (11)</td></tr><tr><td class="halign-left valign-top">/r/Guitar</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">9 (45)</td></tr></tbody></table></div>
<p>VideoLinkBot scans for comments containing supported video links. When it finds one, it scans the discussion that comment belongs to for video links. It then posts the aggregate links it has found to a comment. If it’s already visited this discussion, it will update its existing comment with whatever new links it finds. Video links are sorted by the score of the comment they came from.</p>
<p>If the bot doesn’t see a certain number of links or all the links the bot sees were posted by the same user, the it won’t post a comment. Also, if a discussion has too few or too many comments, this bot will leave it alone.</p>
<p>This provides a useful summary of a wide ranging discussion, in a similar way to <a href="#_tabledresser">tabledresser</a> does for AmA threads. The comments it leaves look like this:</p>
<div class="image-block"><img alt="Screenshot of a comment made by VideoLinkBot, showing the table of aggregated video links, with links to the Source Comment & Video Link, showing the score of each original comment." src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-videolinkbot-example.png"/></div></section>
<section class="doc-section level-2"><h3 id="_meme_transcriber">meme_transcriber</h3><aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>reddit <a href="http://www.reddit.com/r/AdviceAnimals/comments/1gvnk4/quickmeme_is_banned_redditwide_more_inside/">banned quickmeme.com</a> for vote rigging on 22nd June 2013, which <a href="http://www.reddit.com/r/qkme_transcriber/comments/1gvz3z/about_the_banning_of_quickmeme_links/">ended the career of this bots former incarnation, qkme_transcriber</a>.</p></aside>
<div class="dlist"><dl><dt>Purpose</dt><dd>Automatically finds links to meme pics (memegen.com) and provides a plain-text transcript of the content of that meme in a comment, so you don’t have to click through to the meme site to get the ‘joke’. Useful on mobile devices or if the meme site goes down.</dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/meme_transcriber">/u/meme_transcriber</a></li><li><a href="http://www.reddit.com/r/meme_transcriber/">/r/meme_transcriber</a></li><li><a href="http://www.reddit.com/user/qkme_transcriber">/u/qkme_transcriber</a></li><li><a href="http://www.reddit.com/r/qkme_transcriber/">/r/qkme_transcriber</a></li><li><a href="http://www.reddit.com/r/qkme_transcriber/comments/o426k/faq_for_the_qkme_transcriber_bot/">meme_transcriber <span class="caps">FAQ</span></a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>286 link karma</li><li>340,954 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>1 year</dd></dl></div>
<p>This bot tends to turn up in subreddits like <a href="http://www.reddit.com/r/AdviceAnimals/">/r/AdviceAnimals/</a> and post comments that look like this:</p>
<div class="image-block"><img alt="reddit bots meme transcriber bot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-meme-transcriber-bot-example.png"/></div></section>
<section class="doc-section level-2"><h3 id="_ytscreenshotbot">YTScreenShotBot</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Creates a screenshot montage of a YouTube video and posts a link to it, in reply to posts containing YouTube links.</dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/YTScreenShotBot">/u/YTScreenShotBot</a></li></ul></dd><dt>Active SubReddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 22%;"/><col style="width: 42%;"/><col style="width: 36%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/videos</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">420 (2551)</td></tr><tr><td class="halign-left valign-top">/r/pics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">300 (3843)</td></tr><tr><td class="halign-left valign-top">/r/gaming</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">280 (302)</td></tr></tbody></table></div>
<div class="dlist"><dl><dt>Current Karma</dt><dd><ul><li>1 link karma</li><li>15,475 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>25 days</dd></dl></div>
<p>This bot allows you to get a quick overview of the video, just by viewing an image - much quicker than watching the video, especially on mobile devices. This is what it’s comments look like:</p>
<div class="image-block"><img alt="reddit bots ytscreenshotbot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-ytscreenshotbot-example.png"/></div>
<p>and this is what the montage looks like:</p>
<div class="image-block"><img alt="M2XOpjb" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/M2XOpjb.jpg"/></div></section>
<section class="doc-section level-2"><h3 id="_jordanthebrobot">JordanTheBrobot</h3><div class="dlist"><dl><dt>Purpose</dt><dd>A sophisticated Multi-purpose bot that patrols reddit looking for scams, misleading links, mistakes in markup, kindness, flash content, etc…</dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/JordanTheBrobot">/u/JordanTheBrobot</a></li><li><a href="http://jordanthebrobot.com/">JordanTheBrobot <span class="caps">HQ</span></a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>1 link karma</li><li>36,879 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>8 months</dd><dt>Active Subreddits</dt></dl></div>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 31.6831%;"/><col style="width: 36.6336%;"/><col style="width: 31.6833%;"/></colgroup><thead><tr><th class="halign-left valign-top">Subreddit</th><th class="halign-left valign-top">Submissions (karma)</th><th class="halign-left valign-top">Comments (karma)</th></tr></thead><tbody><tr><td class="halign-left valign-top">/r/gaming</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">193 (4614)</td></tr><tr><td class="halign-left valign-top">/r/videos</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">71 (1808)</td></tr><tr><td class="halign-left valign-top">/r/todayilearned</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">36 (221)</td></tr><tr><td class="halign-left valign-top">/r/gonewild</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">32 (34)</td></tr><tr><td class="halign-left valign-top">/r/pics</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">27 (277)</td></tr><tr><td class="halign-left valign-top">/r/AdviceAnimals</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">14 (212)</td></tr><tr><td class="halign-left valign-top">/r/ginger</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">14 (33)</td></tr><tr><td class="halign-left valign-top">/r/Bitcoin</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">13 (80)</td></tr><tr><td class="halign-left valign-top">/r/worldnews</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">13 (68)</td></tr><tr><td class="halign-left valign-top">/r/movies</td><td class="halign-left valign-top">0</td><td class="halign-left valign-top">12 (49)</td></tr><tr><td class="halign-left valign-top">/r/brobot</td><td class="halign-left valign-top">5 (36)</td><td class="halign-left valign-top">3 (3)</td></tr></tbody></table></div>
<p>This bots most user visible function is to detect when people have got the markdown syntax for links the wrong way round (a very common mistake), and if they don’t correct it themselves within a few minutes, leave a reply with the corrected links:</p>
<div class="image-block"><img alt="reddit bots jordanthebrobot example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-jordanthebrobot-example.png"/></div>
<p>It also detects ‘spam/affiliate marketing’ links and leaves a reply warning people:</p>
<div class="quote-block"><blockquote><p><strong>Spam Link</strong></p>
<p>The comment above contains a link to a spam site, click with caution, your clicks will earn a spammer money and give them motivation to continue.</p></blockquote></div>
<p>This bot also has <a href="http://jordanthebrobot.com/moderators">moderator functionality</a>, if you add it as a moderator of a subreddit, it will automatically:</p>
<div class="ulist"><ul><li>Follows all links posted to all subreddits to identify dangerous redirect chains</li><li>Scans comments/submissions/redirect chains for spam domains</li><li>Detects and warns users of mismatched domains in reddit link markup <span class="caps">IE</span>: [<a class="bare" href="http://test.com">http://test.com</a>](<a href="http://not-really-test.com">http://test.com</a>(<a class="bare" href="http://not-really-test.com">http://not-really-test.com</a>])</li><li>Detects and waits 6 minutes to post a fix of mistakes in reddit link markup (for ease of clicking)</li><li>Warns users of unapparent links to flash content</li></ul></div>
<p>It also upvotes the original commenter if it corrects you links and upvotes you if you thank it - which might help it’s popularity. It also has a real time <a href="http://jordanthebrobot.com/">dashboard</a> which lets you see what it’s up to.</p></section>
<section class="doc-section level-2"><h3 id="_serendipitybot_rserendipity">SerendipityBot <span class="amp">&</span> /r/Serendipity</h3><div class="dlist"><dl><dt>Purpose</dt><dd>Cross posts a popular submission from a random subreddit to <a href="http://www.reddit.com/r/Serendipity/">/r/Serendipity</a> every few hours</dd><dt>Home Base</dt><dd><ul><li><a href="http://www.reddit.com/user/serendipitybot">/u/serendipitybot</a></li><li><a href="http://www.reddit.com/r/Serendipity/">/r/Serendipity</a></li></ul></dd><dt>Current Karma</dt><dd><ul><li>37,027 link karma</li><li>2,641 comment karma</li></ul></dd><dt>A Redditor for</dt><dd>2 years</dd><dt>Source Code</dt><dd><a href="https://github.com/umbrae/Serendipity">https://github.com/umbrae/Serendipity</a></dd></dl></div>
<figure class="image-block"><img alt="reddit bots serendipity example" src="https://duncanlock.net/images/posts/a-marvellous-incomplete-compendium-of-reddit-automatons-bots/reddit-bots-serendipity-example.png"/>
<figcaption>Figure 6. Slice of life, reddit style.</figcaption></figure>
<p>I discovered this bot <span class="amp">&</span> subreddit combo while writing this article and it’s quickly become one of my favourites. <a href="http://www.reddit.com/r/Serendipity/">/r/Serendipity</a> is a meta-subreddit meant to broaden the perspective of its subscribers. It chooses a popular post from a completely random subreddit and posts it every few hours, so if you subscribe to it, you get a broad, random, serendipitous sprinkling of great content from across reddit on your front page – often surprising, wonderful things that you would otherwise never have come across. As the sidebar says:</p>
<div class="quote-block"><blockquote><p>If you want to increase your exposure to niche subreddits, or just your perspective on things on the web in general, serendipity might help you do that. But it might not. It’s a bot, after all.</p></blockquote></div>
<p><strong><span class="caps">NB</span></strong>: Occasionally, just by chance, a random post might be <span class="caps">NSFW</span> (Not Safe for Work) or <span class="caps">NSFL</span> (Not Safe for Life - i.e. ugh, wish I could un-see.), but not very often. I asked the bots creator, <a href="http://www.reddit.com/user/umbrae">/u/umbrae</a>, if it did any filtering - this is what he said:</p>
<div class="quote-block"><blockquote><p>It’s actually a bit complicated: It does technically filter out <span class="caps">NSFW</span> subreddits, but does not necessarily filter out <span class="caps">NSFW</span> posts from subreddits that are not marked <span class="caps">NSFW</span>. So you’ll occasionally get a <span class="caps">NSFW</span> post here and there. There are also a few subs that have asked to be opted out for privacy /audience concerns. – <a href="http://www.reddit.com/user/umbrae">/u/umbrae</a>, in <a href="http://www.reddit.com/r/explainlikeimfive/comments/1icm90/eli5_how_do_bots_on_reddit_work_how_are_they/cb3l4av">this comment</a></p></blockquote></div></section></section>
<section class="doc-section level-1"><h2 id="_other_interesting_bots">Other Interesting Bots</h2><p>I don’t have time to cover all the multitude of great bots on reddit - here’s some other useful or fun ones to checkout:</p>
<div class="ulist"><ul><li><a href="http://www.reddit.com/user/SmileBot">SmileBot</a></li><li><a href="http://www.reddit.com/user/DollarSignBot">DollarSignBot</a></li><li><a href="http://www.reddit.com/user/F1-Bot">F1-Bot</a></li><li><a href="http://www.reddit.com/user/smidsy_bot">RideItBot</a></li><li><a href="http://www.reddit.com/user/SimilarImage">SimilarImage</a></li><li><a href="http://www.reddit.com/user/original-finder">original-finder</a></li><li><a href="http://www.reddit.com/user/Australian_Translate">Australian_Translate</a> and his Arch Nemesis: <a href="http://www.reddit.com/user/FIXES_YOUR_COMMENT">FIXES_YOUR_COMMENT</a></li><li><a href="http://www.reddit.com/user/RepostConspiracyBot">RepostConspiracyBot</a></li><li><a href="http://www.reddit.com/user/CaptionBot">CaptionBot</a></li></ul></div>
<p>Another whole <em>category</em> of bots, that I didn’t have time to go into, are Moderator Bots - designed to assist the human moderators of reddit with their ceaseless work, by automating some of the mechanical stuff:</p>
<div class="ulist"><ul><li><a href="http://www.reddit.com/user/automoderator">AutoModeratorBot</a> - very widely used now <span class="amp">&</span> also open source: <a href="https://github.com/Deimos/AutoModerator/wiki/Features">more information here</a>.</li><li><a href="http://www.reddit.com/user/moderator-bot">moderator-bot</a></li><li><a href="http://www.reddit.com/r/atheismbot">atheismbot</a> <span class="amp">&</span> <a href="http://reddit.com/r/atheismbot/wiki/faq">atheismbot <span class="caps">FAQ</span></a></li><li><a href="http://www.reddit.com/u/DeltaBot">DeltaBot</a> is part of the bot moderation team at <a href="http://www.reddit.com/r/changemyview/">/r/changemyview</a>. It adds a special feature to the subreddit that allows users to awards deltas (∆) to each other.</li></ul></div></section>
<section class="doc-section level-1"><h2 id="_ex_bots">Ex-Bots?</h2><p>Some interesting bots who seem to be ex-bots – or maybe they’re just resting:</p>
<div class="ulist"><ul><li><a href="http://www.reddit.com/user/Meta_Bot">Meta_Bot</a></li><li><a href="http://www.reddit.com/user/canhekickit">canhekickit</a></li><li><a href="http://www.reddit.com/user/QualityEnforcer">QualityEnforcer</a></li><li><a href="http://www.reddit.com/user/PoliticalBot">PoliticalBot</a> <span class="amp">&</span> <a href="http://www.reddit.com/r/AnalyzingReddit">AnalyzingReddit</a></li><li><a href="http://www.reddit.com/user/Match-Thread-Bot">Match-Thread-Bot</a></li><li><a href="http://www.reddit.com/user/linkfixerbot">LinkFixerBot</a></li><li><a href="http://www.reddit.com/user/tweet_poster">tweet_poster</a></li><li><a href="http://www.reddit.com/user/Karmangler">Karmangler</a></li><li><a href="http://www.reddit.com/user/autotldr">autotldr</a></li><li><a href="http://www.reddit.com/user/CONGRATS_GUY">CONGRATS_GUY</a></li><li><a href="http://www.reddit.com/r/qkme_transcriber/comments/1gvz3z/about_the_banning_of_quickmeme_links/">qkme_transcriber</a></li></ul></div>
<hr/>
<p>Know of any more interesting <span class="amp">&</span> fun reddit bots? Let me know in the comments…</p>
<hr/>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References</h3></section></section><section aria-label="Footnotes" class="footnotes" role="doc-endnotes"><hr/><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote"><a href="http://www.reddit.com/about/">About reddit, including some mind boggling statistics</a>. <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">How many bots? No one really knows. <a href="https://praw.readthedocs.org/en/latest/">How to create a reddit bot</a>. This being reddit, there’s <a href="http://www.reddit.com/r/botwatch">a community</a> to keep an eye on them, too - and <a href="http://www.reddit.com/r/TheoryOfReddit/">/r/TheoryOfReddit</a> do <a href="http://www.reddit.com/r/TheoryOfReddit/comments/187n3n/reddit_has_bots_but_what_kinds_of_bots_are_there/">sometimes</a> <a href="http://www.reddit.com/r/TheoryOfReddit/comments/1586yk/should_reddit_regulate_bots/">discuss</a> bots. Well, <a href="http://www.reddit.com/r/TheoryOfReddit/comments/m5t1s/a_worrying_trend_for_reddits_bots/">actually</a> they <a href="http://www.reddit.com/r/IAmA/comments/kglw8/we_are_the_creators_of_the_automated_bots_on/">talk</a> <a href="http://www.reddit.com/r/TheoryOfReddit/comments/k7xjw/lets_talk_about_bots/">about</a> bots <a href="http://www.reddit.com/r/TheoryOfReddit/search?q=bot&restrict_sr=on">quite a lot</a>. <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_3" role="doc-endnote">This is mostly quoted from the excellent qkme_transcriber bot’s <span class="caps">FAQ</span>, <a href="http://www.reddit.com/r/qkme_transcriber/comments/o426k/faq_for_the_qkme_transcriber_bot/">here</a>. <a class="footnote-backref" href="#_footnoteref_3" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_4" role="doc-endnote">*<span class="caps">API</span>*: An agreed way for one piece of software to talk to another. Often consists of functions you can call with parameters, that return different peices of information - or perform different actions - depending on the value of the parameters. In the case of websites, the functions map to <span class="caps">URL</span>’s - pages that you can request, with the parameters on the end of the <span class="caps">URL</span>. *Why does reddit have an <span class="caps">API</span>?* Well, people would find a way to get the same information somehow - often by brute force (acting like a very fast human making lots of requests) - which puts more strain on reddit’s servers than just giving the data out in one go, on request - it also means that they get to set the rules when they make the <span class="caps">API</span>. <a class="footnote-backref" href="#_footnoteref_4" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_5" role="doc-endnote"><a href="http://www.reddit.com/r/circlejerk/top/">/r/circlejerk</a> is a subreddit dedicated entirely to reddit satire. It’s full of ‘parodies’ of ‘karma whoring’ posts and ‘parodies’ of endless pun threads. The thought that they have rigorous standards and actually kick people out for breaking them is almost funny in itself. <a class="footnote-backref" href="#_footnoteref_5" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_6" role="doc-endnote"><a href="http://www.reddit.com/r/redditdev/comments/1ixqu0/praw_where_do_you_all_host_your_pythonbased_bots/">/r/redditdev/ thread: Where do you all host your python-based bots?</a> - turns out YTScreenhostBot is hosted on an old laptop. <a class="footnote-backref" href="#_footnoteref_6" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_7" role="doc-endnote"><a href="http://www.reddit.com/r/TheoryOfReddit/comments/tiqqg/how_easily_could_a_computer_program_emulate_the/">How easily could a computer program emulate the average reddit commenter?</a> <a class="footnote-backref" href="#_footnoteref_7" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_8" role="doc-endnote">Internet Points! reddit has a system called <a href="http://www.reddit.com/wiki/faq#wiki_what_is_that_number_next_to_usernames.3F_and_what_is_karma.3F">Karma</a> : “The number next to a username is called that user’s “karma.” It reflects how much good the user has done for the reddit community. The best way to gain karma is to submit links that other people like and vote for.” <a class="footnote-backref" href="#_footnoteref_8" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_9" role="doc-endnote"><a href="http://en.wikipedia.org/wiki/Haiku">Haiku</a>: In English, Haiku are traditionally three line verses, each line having 5, 7 <span class="amp">&</span> 5 syllables respectively. <a class="footnote-backref" href="#_footnoteref_9" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>The Smart Guide to Stack Overflow: Zero to Hero2013-06-14T15:03:28-07:002021-06-10T23:42:55-07:00Duncan Locktag:duncanlock.net,2013-06-14:/blog/2013/06/14/the-smart-guide-to-stack-overflow-zero-to-hero/<figure class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stackoverflow-logo-is-made-of-people-diagram.png" alt="stackoverflow logo is made of people diagram">
<figcaption>Figure 1. StackOverflow is made of people - lots and lots and lots of people.</figcaption></figure>
<p>Seemingly like everything else in the world, Stack Overflow is getting more <span class="amp">&</span> more competitive as time passes. Gaining a good reputation - and even finding good questions to answer - is becoming harder and harder, as more and more people compete for less and less unanswered questions.</p>
<p>This is <em>fantastic</em> if you want to find an answer - but makes it much harder for newcomers to get started and build a reputation on the site.</p>
<p>This guide will help you jump-start your reputation on StackOverflow and help you get more out of the site.</p>
<section class="doc-section level-1"><h2 id="_how_to_find_questions_you_can_answer">How to find questions you can answer</h2><section class="doc-section level-2"><h3 id="_ignored_favourite_tags">Ignored <span class="amp">&</span> Favourite Tags</h3><figure class="image-block align-right"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stack-overflow-ignored-tags.png" alt="Screenshot of part of my (very long) ignored tags list from Stack Overflow">
<figcaption>Figure 2. I don’t know jack: Screenshot of part of my (long) ignored tags list from Stack Overflow</figcaption></figure>
<p>Use the <a href="http://stackoverflow.com/unanswered/tagged/?tab=noanswers">no answers</a> page, the …</p></section></section><figure class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stackoverflow-logo-is-made-of-people-diagram.png" alt="stackoverflow logo is made of people diagram">
<figcaption>Figure 1. StackOverflow is made of people - lots and lots and lots of people.</figcaption></figure>
<p>Seemingly like everything else in the world, Stack Overflow is getting more <span class="amp">&</span> more competitive as time passes. Gaining a good reputation - and even finding good questions to answer - is becoming harder and harder, as more and more people compete for less and less unanswered questions.</p>
<p>This is <em>fantastic</em> if you want to find an answer - but makes it much harder for newcomers to get started and build a reputation on the site.</p>
<p>This guide will help you jump-start your reputation on StackOverflow and help you get more out of the site.</p>
<section class="doc-section level-1"><h2 id="_how_to_find_questions_you_can_answer">How to find questions you can answer</h2><section class="doc-section level-2"><h3 id="_ignored_favourite_tags">Ignored <span class="amp">&</span> Favourite Tags</h3><figure class="image-block align-right"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stack-overflow-ignored-tags.png" alt="Screenshot of part of my (very long) ignored tags list from Stack Overflow">
<figcaption>Figure 2. I don’t know jack: Screenshot of part of my (long) ignored tags list from Stack Overflow</figcaption></figure>
<p>Use the <a href="http://stackoverflow.com/unanswered/tagged/?tab=noanswers">no answers</a> page, the <a href="http://stackoverflow.com/unanswered/tagged/?tab=newest">newest unanswered page</a> and the <a href="http://stackoverflow.com/questions?sort=featured">bounties pages</a>, but make sure you add plenty of Ignored Tags. Whenever you see a question that you don’t know anything about, add it’s tags to your Ignored Tags list. This fades questions with these tags, making it much easier to skim through the list finding things that you <em>can</em> answer.</p>
<p>Don’t be afraid to add lots of ignored tags - it’s not an admission of failure - there will always be lots of things that you aren’t an expert on.</p>
<p>Add some favourite tags too - this highlights questions with these tags, again, helping you to sort the wheat from the chaff.</p></section>
<section class="doc-section level-2"><h3 id="_wildcards">Wildcards</h3><div class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stack-overflow-ignored-tags-add-with-wildcard.png" alt="stack overflow ignored tags add with wildcard"></div>
<p>Both Ignored and Favourite Tags can use wild cards - just put an asterisk at the end of the tag when you enter it - so <code>xcode*</code> would match both <code>xcode</code> and <code>xcode4.5</code>, for example. This significantly reduces the number of tags you need.</p></section>
<section class="doc-section level-2"><h3 id="_specialize_follow_tags_automate">Specialize, Follow Tags <span class="amp">&</span> Automate</h3><p>You can also follow a single tag - one of your Favourite tags, for example - just click on it in your sidebar and you’ll see a <a href="http://stackoverflow.com/questions/tagged/mysql%2A">filtered version of the Questions pages, just for that tag</a>. This also works for wildcards, so clicking on a wildcard tag in your sidebar will filter by all matching tags.</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stack-overflow-follow-tags-feed-with-wildcard.png" alt="stack overflow follow tags feed with wildcard"></div>
<p>At the bottom of each of these filtered lists is a link to an Atom Feed <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a> that you can subscribe to in a News/Feed Reader <a class="footnote-ref" id="_footnoteref_2" href="#_footnote_2" title="View footnote 2" role="doc-noteref">[2]</a> - this will alert you whenever new questions are posted with that tag.</p>
<p>You can also use <a href="https://ifttt.com/recipes/search?q=stackoverflow"><span class="caps">IFTTT</span> to automatically notify</a> <a class="footnote-ref" id="_footnoteref_3" href="#_footnote_3" title="View footnote 3" role="doc-noteref">[3]</a> you when new questions appear, when your reputation changes, etc - it can even send you an <span class="caps">SMS</span>, so you can jump in immediately.</p></section></section>
<section class="doc-section level-1"><h2 id="_mercenary_reputation_building_101">Mercenary Reputation Building 101</h2><p>Most of these reputation building techniques are win-win, thanks to Stack Overflow’s masterful gamification <a class="footnote-ref" id="_footnoteref_4" href="#_footnote_4" title="View footnote 4" role="doc-noteref">[4]</a> architecture - but some of them are a little bit…​ mercenary:</p>
<section class="doc-section level-2"><h3 id="_answers_in_the_comments_are_just_asking_for_it">Answers in the comments are just asking for it</h3><p>For some reason, people often answer questions in the comments, rather than submitting an actual Answer - this is wrong for lots of reasons:</p>
<div class="ulist"><ul><li>It makes the answers hard to find for other people <span class="amp">&</span> adds to the noise on the site.</li><li>It means that an answer can never be accepted, so other people won’t be able to tell if the question has an answer - and the question will appear in the Unanswered lists forever.</li><li>People get no credit for answers supplied in the comments</li><li>Formatting and space is intentionally very limited in comments - to try to dissuade people posting answers there.</li></ul></div>
<p>If people are stupid enough to do this, then they deserve to be punished (a little tiny bit). Copy their answer out of the comment into a real Answer. Expand on it in the extra space you now have, improve the formatting, add example code, etc…​ - and wait for the reputation points to roll in.</p></section>
<section class="doc-section level-2"><h3 id="_answer_questions_from_experienced_users">Answer questions from experienced users</h3><figure class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/screenshot-13-06-06-07-43-18-pm.png" alt="screenshot 13 06 06 07 43 18 pm">
<figcaption>Figure 3. Except this guy.</figcaption></figure>
<p>Everything else being equal, answer questions asked by users with higher reputation. They know how the site works, they know to accept and upvote answers. New users often don’t, or they’ll ask a question and never some back - so it’s much more likely that you’ll get no reward for answering their questions.</p>
<p>New users also tend to ask lower quality questions, which are harder to answer well - and get closed more often - than questions from experienced users.</p>
<p>This is a little hard on new users, so if they’ve asked a well articulated, valid question and have a real looking username, I would suggest taking a chance and answering the question anyway. Even if the asker never comes back, you can still get upvotes from other users.</p>
<p>On the other hand…​</p></section>
<section class="doc-section level-2"><h3 id="_encourage_new_users">Encourage New Users</h3><figure class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stackoverflow-upvotes-accept-encourage-users.png" alt="stackoverflow upvotes accept encourage users">
<figcaption>Figure 4. These two techniques, used together, are very effective.</figcaption></figure>
<p>As <a href="#_article_comments_section">ray pointed out in the comments</a>, perhaps I was a little bit hard on new users.</p>
<p>So, you spend time researching and writing a solid, well written, correct answer to a question and…​ crickets. It’s either the only answer, the only good one, or it’s gets upvoted but not accepted. What now?</p>
<p>Here are two ways to encourage users (new or not) to upvote and accept your answers to their questions:</p>
<div class="dlist"><dl><dt>Leave a comment</dt><dd>Leave them a comment on their question, asking if it worked, if they had any issues with the answer and asking them to accept the answer if it worked for them - so that other users can benefit: from knowing that the answer works and by having the question marked as Answered.</dd><dt>Upvote their question</dt><dd>As Ray said: “I tend to wait and get some up votes, by that time the new user is used to <span class="caps">SO</span> so I up vote the question, 10 times of 10 he always accepts my answer and adds an upvote, 25 points collected by being patient with new users like me.”</dd></dl></div></section>
<section class="doc-section level-2"><h3 id="_speculate_to_accumulate_answer_lots_of_questions_well">Speculate to Accumulate: Answer lots of questions, well</h3><p>This seems obvious, but the more questions you answer well, the more reputation you’ll gain - but not just at the time you answer: forever. Good answers keep building reputation over time as new people discover them and upvote them - and the more of your answers are out there, the more you’ll gain from this on an ongoing basis.</p>
<figure class="image-block"><a class="image" href="http://stackoverflow.com/questions/2675323/mysql-load-null-values-from-csv-data/5968530#5968530"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/screenshot-13-06-06-07-27-10-pm.png" alt="screenshot 13 06 06 07 27 10 pm"></a>
<figcaption>Figure 5. This answer was posted in May 2011, this screenshot was taken in June 2013.</figcaption></figure>
<p>Once you’ve got the top voted answer on a popular question, you will gain occasional upvotes and reputation from it without you actively doing anything. The more of this you have and the more popular those questions and answers, the more you gain - with <a href="http://stackoverflow.com/users/1288/bill-the-lizard?tab=reputation">top users</a> sometimes hitting the daily reputation cap of +200 without doing anything.</p></section>
<section class="doc-section level-2"><h3 id="_come_back_and_improve_your_popular_answers">Come back and improve your popular answers</h3><p>If you find that one of your answers keeps receiving upvotes over time, then come back and improve on it. Edit your answer - improve the formatting, add better example code and answer any comments people have left - by improving the answer to address them. This increases the amount of upvotes that your improved answer will get over time and improves the quality of the site overall.
It also bumps that question back up in search results and lists, making it more visible, increasing the likelihood of upvotes, and so on.</p>
<p>However - don’t do this <em>too</em> often. Only make edits that are worthwhile and add value to the answer - if you edit your answers too much, they’ll become <a href="http://meta.stackoverflow.com/questions/11740/what-are-community-wiki-posts">Community Wiki posts</a> and stop generating reputation altogether.</p></section>
<section class="doc-section level-2"><h3 id="_get_in_first">Get in First</h3><p>Being the first answer is often surprisingly important. There are lots of other people looking through the unanswered questions list for questions to answer - as soon as a question has an answer it disappears from this list. Being the first correct answer to a question also makes it more likely that other people visiting the question will upvote your answer and move on, looking for something else to answer. Answers are sorted by votes, so the answer with the first upvote will move to the top, thus getting more attention and re-enforcing the cycle – answers with an early lead will often maintain it.</p>
<p>So, if you see a question that you know the answer to off the top of your head, answer it immediately. Get the gist of the answer down and submit it - don’t spend too long writing this first draft answer, or someone else will beat you to it. Then, read through your answer, think about it some more and edit it - expanding on your answer, adding more detail, improving it with examples and Markdown formatting.</p>
<p>Quick answers are also good for the asker - they get the answer they need quickly and can start working on their solution - and perhaps making follow-up comments while you’re further polishing your answer.</p></section>
<section class="doc-section level-2"><h3 id="_preferentially_answer_questions_with_bounties">Preferentially answer questions with bounties</h3><p>Again, obvious - questions with <a href="http://stackoverflow.com/helpcenter/bounty">bounties</a> give you the bounty as reputation if you post the accepted (or highest voted) answer.</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/screenshot-13-06-06_07-12-23-pm.png" alt="screenshot 13 06 06 07 12 23 pm"></div>
<p>Use the <a href="http://stackoverflow.com/questions?pagesize=50&sort=featured">Featured list</a> to see all questions with bounties. Your ignored and favourite tags work here too.</p>
<p>Even if the person who places the bounty never bothers to come back and award it - half of it will get awarded to the highest voted answer (created after the bounty started with at least 2 upvotes) when the bounty closes. This means that you’re only guaranteed to land the bounty if you can get the top spot, so answering questions with an outstanding bounty - but several existing answers with lots of votes - generally isn’t such a good investment of time. Unless you think you can provide an answer that’s sufficiently good to beat the existing ones before the bounty closes, don’t bother.</p></section>
<section class="doc-section level-2"><h3 id="_getting_badges_earning_valuable_flair">Getting Badges: Earning Valuable Flair</h3><figure class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/screenshot-13-06-06_07-14-59-pm.png" alt="screenshot 13 06 06 07 14 59 pm">
<figcaption>Figure 6. Ooooh look - pieces of valuable flair™</figcaption></figure>
<p>You will accumulate badges in the course of using the site, but there are ways to increase your accumulation rate slightly without going out of your way too much.</p>
<p>Preferentially answering older, unanswered questions is a good way to pick up <a href="http://stackoverflow.com/badges/17/necromancer?userid=259698">Necromancer</a>, <a href="http://stackoverflow.com/badges/837/revival?userid=259698">Revival</a> and <a href="http://stackoverflow.com/badges/1287/excavator?userid=259698">Excavator</a> badges - and using ignored tags is a great way to filter out the noise in the No Answers list, allowing you to quickly zip back to the older unanswered questions that you can answer. There’s also <em>much less</em> competition to answer these poor, neglected questions than there is for new questions, so you can answer at your leisure, taking your time to craft the perfect answer for the ages.</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stackoverflow-sharing-a-link.png" alt="Screenshot of the Sharing buttons at the bottom left of a Question."></div>
<p>Another simple win-win way to acquire badges is by sharing your questions <span class="amp">&</span> answers with your friends, using the sharing buttons on the site. This is a double whammy - your question or answer will get more exposure, so more upvotes - and you can <em>also</em> get badges…​</p>
<figure class="image-block align-right"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stackoverflow-sharing-badges.png" alt="Screenshots of the StackOverflow badges for sharing links.">
<figcaption>Figure 7. StackOverflow’s Sharing badges</figcaption></figure>
<p>The <a href="http://stackoverflow.com/badges/260/announcer">Announcer</a>, <a href="http://stackoverflow.com/badges/261/booster">Booster</a> <span class="amp">&</span> <a href="http://stackoverflow.com/badges/262/publicist">Publicist</a> badges are awarded for sharing a link to a question that was visited by 25, 300 or 1000 unique visitors, respectively - and they can all be awarded multiple times.</p>
<p>See <a href="#_answer_your_own_questions">Answer Your Own Questions</a>, below for another easy to earn badge.</p></section></section>
<section class="doc-section level-1"><h2 id="_ask_good_questions">Ask Good Questions</h2><p>You can also get reputation (and badges) for <em>asking</em> questions: +5 for each upvote your question gets.</p>
<p>How to ask good questions? Questions that attract good answers - and upvotes? The <a href="http://stackoverflow.com/helpcenter/asking">official guide is here</a>. In addition to this, my tips for good questions are:</p>
<section class="doc-section level-2"><h3 id="_search_first">Search first</h3><p>Someone has almost certainly asked your question before and the answer is just there waiting for you. Search with Google <span class="amp">&</span> directly on Stack Overflow.</p></section>
<section class="doc-section level-2"><h3 id="_think_before_you_post">Think before you post</h3><p>Don’t just ask questions for the sake of it - or for the reputation. Ask when you’re <em>genuinely</em> stuck. Try to solve the problem yourself - but if you really can’t, ask. Mention your attempted solutions in the question, so that people know what you’ve already tried and eliminated.</p></section>
<section class="doc-section level-2"><h3 id="_explain_carefully">Explain carefully</h3><p>Carefully explain your problem, in detail, so that someone without any prior knowledge of your situation can understand the problem. They’re not telepathic - you need to explain yourself succinctly and thoughtfully if you want a good answer.</p></section>
<section class="doc-section level-2"><h3 id="_include_a_relevant_simplified_example">Include a relevant simplified example</h3><p>Boil your problem down to its essence and include a simplified example - with any required code and data - in your question. Try and make this as short as possible without leaving out anything essential.</p>
<p>A working example, using <a href="http://jsfiddle.net/">jsfiddle</a>, <a href="http://sqlfiddle.com/">sqlfiddle</a>, <a href="http://rubyfiddle.com/">rubyfiddle</a>, etc…​ is the gold standard. Put the simplified example code into your question as normal, but also upload it to the relevant *fiddle site and add the link to your question.</p></section>
<section class="doc-section level-2"><h3 id="_use_markdown_formatting">Use Markdown formatting</h3><p>This goes for both asking questions and answering them. Stack Overflow <a href="http://stackoverflow.com/editing-help">supports Markdown for formatting your posts</a> - <em>use it</em>! It will make your questions easier to read and understand, you’ll get more upvotes and better answers.</p></section>
<section class="doc-section level-2"><h3 id="_read_before_posting_then_read_it_again_afterwards">Read before posting, then read it again afterwards</h3><p>Read your question through a few times before posting. Make sure that it’s well phrased, well formatted and spelt correctly. Make sure that your example code and data is clear and concise and includes everything you would need to reproduce the problem.</p>
<p>Once you’ve posted it, read the live version and edit out the mistakes you missed before posting.</p></section></section>
<section class="doc-section level-1"><h2 id="_answer_your_own_questions">Answer your own Questions</h2><p>In the unlikely event that you can’t get any help from StackOverflow initially - but later figure out the solution yourself - post both the question and the answer at the same time. As <a href="http://balpha.de/">balpha</a> said in the <a href="#_article_comments_section">comments section</a>:</p>
<p>If you’ve had a hard or interesting problem for which there’s nothing on Stack Overflow yet, and you have eventually managed to solve it yourself: Ask <em>and answer</em> the question. Someone else is bound to be having the same problem, and you already did the hard work. The “ask question” interface has a checkbox that lets you submit an answer alongside with the question. And if you’ve already asked the question, and then <em>later</em> managed to solve the problem: Go ahead, answer your own question.</p>
<p>Not only can you spare the next person with the same issue having to figure it out all over again - you also have a chance to get an upvote from them on both the question and the answer, for a total of 15 reputation!</p>
<p>Reputation Bonanza!</p>
<p>If you later figure out the answer to one of your questions - or figure out a <em>better</em> answer, or a new solution becomes available, come back and tell everyone by either answering - or adding an answer - to your own question: everyone wins.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/the-smart-guide-to-stack-overflow-zero-to-hero/stackoverflow-self-learner-badge.png" alt="Screenshot of the Self Learner Badge from StackOverflow">
<figcaption>Figure 8. Answered your own question with score of 3 or more.</figcaption></figure>
<p>This is <a href="http://blog.stackoverflow.com/2011/07/its-ok-to-ask-and-answer-your-own-questions/">offically encouraged</a> - there are even badges for doing it, so Ask and Answer away!</p>
<p>If you’ve got any tips or advice I’ve missed, I’d love to hear about them in the comments below.</p>
<hr>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References</h3></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">*Atom Feeds* (like <span class="caps">RSS</span> Feeds) can be used to allow users to subscribe to updates from a website. <a href="http://en.wikipedia.org/wiki/Atom_(standard)">Wikipedia Atom Article…​</a> <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">A *Feed Reader* is a piece of software (Desktop, Mobile or Web based) that allows users to collect/aggregate and read their Feeds, manage subscriptions and send notifications. <a href="http://en.wikipedia.org/wiki/Feed_reader">Wikipedia Feed Reader Article…​</a> <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_3" role="doc-endnote">*<span class="caps">IFTTT</span>* enables you to create and share “recipes” that fit the simple statement: “if this then that”. The “this” part of a recipe is a trigger. Some example triggers are “I’m tagged in a photo on Facebook” or “I check in on Foursquare.” The “that” part of a recipe is an action. Some example actions are “send me a text message” or “create a status message on Facebook.”. <a href="http://en.wikipedia.org/wiki/IFTTT">Wikipedia <span class="caps">IFTTT</span> Article…​</a> <a class="footnote-backref" href="#_footnoteref_3" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_4" role="doc-endnote">*Gamification* is the use of game thinking and game mechanics in a non-game context in order to engage users and solve problems. <a href="http://en.wikipedia.org/wiki/Gamification">Wikipedia Gamification Article…​</a> <a class="footnote-backref" href="#_footnoteref_4" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>How to switch to Compton for beautiful tear free compositing in XFCE2013-06-07T00:05:59-07:002021-06-11T20:03:53-07:00Duncan Locktag:duncanlock.net,2013-06-07:/blog/2013/06/07/how-to-switch-to-compton-for-beautiful-tear-free-compositing-in-xfce/<p>I switched my <span class="caps">XFCE</span> machines over to use Compton for window compositing today - and it’s a noticeable improvement.</p>
<p>A compositor glues your stacks of windows together to form the final image that you see on screen. It’s responsible for any fancy effects like drop-shadows, as well drawing windows while dragging, resizing and minimizing or maximizing them. <a class="footnote-ref" href="#_footnote_1" id="_footnoteref_1" role="doc-noteref" title="View footnote 1">[1]</a></p>
<p>Compton does this <em>beautifully</em>. It does one thing and it does it well. It provides glassy smooth, tear free compositing and supports a few tasteful effects - drop shadows, fades and transparency.</p>
<p>I stumbled across this while searching for something else and found a great guide in the NeoWin forums <a class="footnote-ref" href="#_footnote_2" id="_footnoteref_2" role="doc-noteref" title="View footnote 2">[2]</a>, where most of this info comes from, so thanks ViperAFK.</p>
<section class="doc-section level-1"><h2 id="_switch_off_the_existing_compositor">Switch off the existing compositor</h2><div class="image-block align-right"><img alt="Screenshot of the XFCE Applications menu, with the Settings Manager highlighted." src="https://duncanlock.net/images/posts/how-to-switch-to-compton-for-beautiful-tear-free-compositing-in-xfce/xfce-applications-menu-settings-manager.png"/></div>
<p>You’ll need to switch of any existing compositing you’ve got running, otherwise this won …</p></section><p>I switched my <span class="caps">XFCE</span> machines over to use Compton for window compositing today - and it’s a noticeable improvement.</p>
<p>A compositor glues your stacks of windows together to form the final image that you see on screen. It’s responsible for any fancy effects like drop-shadows, as well drawing windows while dragging, resizing and minimizing or maximizing them. <a class="footnote-ref" href="#_footnote_1" id="_footnoteref_1" role="doc-noteref" title="View footnote 1">[1]</a></p>
<p>Compton does this <em>beautifully</em>. It does one thing and it does it well. It provides glassy smooth, tear free compositing and supports a few tasteful effects - drop shadows, fades and transparency.</p>
<p>I stumbled across this while searching for something else and found a great guide in the NeoWin forums <a class="footnote-ref" href="#_footnote_2" id="_footnoteref_2" role="doc-noteref" title="View footnote 2">[2]</a>, where most of this info comes from, so thanks ViperAFK.</p>
<section class="doc-section level-1"><h2 id="_switch_off_the_existing_compositor">Switch off the existing compositor</h2><div class="image-block align-right"><img alt="Screenshot of the XFCE Applications menu, with the Settings Manager highlighted." src="https://duncanlock.net/images/posts/how-to-switch-to-compton-for-beautiful-tear-free-compositing-in-xfce/xfce-applications-menu-settings-manager.png"/></div>
<p>You’ll need to switch of any existing compositing you’ve got running, otherwise this won’t work. Unless you know differently, this will be the default one, built into the <span class="caps">XFCE</span> window manager, xfwm4.</p>
<p>To switch this off, go into the Applications menu and click ‘Settings Manager’:</p>
<p>Then click ‘Window Manager Tweaks’, then the ‘Compositor’ tab, and un-tick the ‘Enable Display Compositing’ box:</p>
<div class="image-block"><img alt="Screenshot of the XFCE Settings Manager - Window Manager Tweaks window, with 'Enable Display Compositing un-ticked'" src="https://duncanlock.net/images/posts/how-to-switch-to-compton-for-beautiful-tear-free-compositing-in-xfce/xfce-settings-manager-window-manager-tweaks-disable-compositing.png"/></div>
<p>Once you’ve switched off any existing compositor, you can install Compton.</p></section>
<section class="doc-section level-1"><h2 id="_install_compton">Install Compton</h2><p>To do that, run the following commands in a terminal window. These will add the official Compton <span class="caps">PPA</span> <a class="footnote-ref" href="#_footnote_3" id="_footnoteref_3" role="doc-noteref" title="View footnote 3">[3]</a> and then install it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-add-repository ppa:richardgv/compton
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get update
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>compton</code></pre></div></section>
<section class="doc-section level-1"><h2 id="_configure_compton">Configure Compton</h2><p>Once it’s installed, create a text file in <code>~/.config/</code> called <code>compton.conf</code> with the following contents:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="conf"><span class="c">#################################
#
# Backend
#
#################################
</span>
<span class="c"># Backend to use: "xrender" or "glx".
# GLX backend is typically much faster but depends on a sane driver.
</span><span class="n">backend</span> = <span class="s2">"glx"</span>;
<span class="c">#################################
#
# GLX backend
#
#################################
</span>
<span class="n">glx</span>-<span class="n">no</span>-<span class="n">stencil</span> = <span class="n">true</span>;
<span class="c"># GLX backend: Copy unmodified regions from front buffer instead of redrawing them all.
# My tests with nvidia-drivers show a 10% decrease in performance when the whole screen is modified,
# but a 20% increase when only 1/4 is.
# My tests on nouveau show terrible slowdown.
# Useful with --glx-swap-method, as well.
</span><span class="n">glx</span>-<span class="n">copy</span>-<span class="n">from</span>-<span class="n">front</span> = <span class="n">false</span>;
<span class="c"># GLX backend: Use MESA_copy_sub_buffer to do partial screen update.
# My tests on nouveau shows a 200% performance boost when only 1/4 of the screen is updated.
# May break VSync and is not available on some drivers.
# Overrides --glx-copy-from-front.
# glx-use-copysubbuffermesa = true;
</span>
<span class="c"># GLX backend: Avoid rebinding pixmap on window damage.
# Probably could improve performance on rapid window content changes, but is known to break things on some drivers (LLVMpipe).
# Recommended if it works.
# glx-no-rebind-pixmap = true;
</span>
<span class="c"># GLX backend: GLX buffer swap method we assume.
# Could be undefined (0), copy (1), exchange (2), 3-6, or buffer-age (-1).
# undefined is the slowest and the safest, and the default value.
# copy is fastest, but may fail on some drivers,
# 2-6 are gradually slower but safer (6 is still faster than 0).
# Usually, double buffer means 2, triple buffer means 3.
# buffer-age means auto-detect using GLX_EXT_buffer_age, supported by some drivers.
# Useless with --glx-use-copysubbuffermesa.
# Partially breaks --resize-damage.
# Defaults to undefined.
</span><span class="n">glx</span>-<span class="n">swap</span>-<span class="n">method</span> = <span class="s2">"undefined"</span>;
<span class="c">#################################
#
# Shadows
#
#################################
</span>
<span class="c"># Enabled client-side shadows on windows.
</span><span class="n">shadow</span> = <span class="n">true</span>;
<span class="c"># Don't draw shadows on DND windows.
</span><span class="n">no</span>-<span class="n">dnd</span>-<span class="n">shadow</span> = <span class="n">true</span>;
<span class="c"># Avoid drawing shadows on dock/panel windows.
</span><span class="n">no</span>-<span class="n">dock</span>-<span class="n">shadow</span> = <span class="n">true</span>;
<span class="c"># Zero the part of the shadow's mask behind the window. Fix some weirdness with ARGB windows.
</span><span class="n">clear</span>-<span class="n">shadow</span> = <span class="n">true</span>;
<span class="c"># The blur radius for shadows. (default 12)
</span><span class="n">shadow</span>-<span class="n">radius</span> = <span class="m">5</span>;
<span class="c"># The left offset for shadows. (default -15)
</span><span class="n">shadow</span>-<span class="n">offset</span>-<span class="n">x</span> = -<span class="m">5</span>;
<span class="c"># The top offset for shadows. (default -15)
</span><span class="n">shadow</span>-<span class="n">offset</span>-<span class="n">y</span> = -<span class="m">5</span>;
<span class="c"># The translucency for shadows. (default .75)
</span><span class="n">shadow</span>-<span class="n">opacity</span> = <span class="m">0</span>.<span class="m">5</span>;
<span class="c"># Set if you want different colour shadows
# shadow-red = 0.0;
# shadow-green = 0.0;
# shadow-blue = 0.0;
</span>
<span class="c"># The shadow exclude options are helpful if you have shadows enabled. Due to the way compton draws its shadows, certain applications will have visual glitches
# (most applications are fine, only apps that do weird things with xshapes or argb are affected).
# This list includes all the affected apps I found in my testing. The "! name~=''" part excludes shadows on any "Unknown" windows, this prevents a visual glitch with the XFWM alt tab switcher.
</span><span class="n">shadow</span>-<span class="n">exclude</span> = [
<span class="s2">"! name~=''"</span>,
<span class="s2">"name = 'Notification'"</span>,
<span class="s2">"name = 'Plank'"</span>,
<span class="s2">"name = 'Docky'"</span>,
<span class="s2">"name = 'Kupfer'"</span>,
<span class="s2">"name = 'xfce4-notifyd'"</span>,
<span class="s2">"name *= 'VLC'"</span>,
<span class="s2">"name *= 'compton'"</span>,
<span class="s2">"name *= 'Chromium'"</span>,
<span class="s2">"name *= 'Chrome'"</span>,
<span class="s2">"name *= 'Firefox'"</span>,
<span class="s2">"class_g = 'Conky'"</span>,
<span class="s2">"class_g = 'Kupfer'"</span>,
<span class="s2">"class_g = 'Synapse'"</span>,
<span class="s2">"class_g ?= 'Notify-osd'"</span>,
<span class="s2">"class_g ?= 'Cairo-dock'"</span>,
<span class="s2">"class_g ?= 'Xfce4-notifyd'"</span>,
<span class="s2">"class_g ?= 'Xfce4-power-manager'"</span>
];
<span class="c"># Avoid drawing shadow on all shaped windows (see also: --detect-rounded-corners)
</span><span class="n">shadow</span>-<span class="n">ignore</span>-<span class="n">shaped</span> = <span class="n">false</span>;
<span class="c">#################################
#
# Opacity
#
#################################
</span>
<span class="n">menu</span>-<span class="n">opacity</span> = <span class="m">1</span>;
<span class="n">inactive</span>-<span class="n">opacity</span> = <span class="m">1</span>;
<span class="n">active</span>-<span class="n">opacity</span> = <span class="m">1</span>;
<span class="n">frame</span>-<span class="n">opacity</span> = <span class="m">1</span>;
<span class="n">inactive</span>-<span class="n">opacity</span>-<span class="n">override</span> = <span class="n">false</span>;
<span class="n">alpha</span>-<span class="n">step</span> = <span class="m">0</span>.<span class="m">06</span>;
<span class="c"># Dim inactive windows. (0.0 - 1.0)
# inactive-dim = 0.2;
# Do not let dimness adjust based on window opacity.
# inactive-dim-fixed = true;
# Blur background of transparent windows. Bad performance with X Render backend. GLX backend is preferred.
# blur-background = true;
# Blur background of opaque windows with transparent frames as well.
# blur-background-frame = true;
# Do not let blur radius adjust based on window opacity.
</span><span class="n">blur</span>-<span class="n">background</span>-<span class="n">fixed</span> = <span class="n">false</span>;
<span class="n">blur</span>-<span class="n">background</span>-<span class="n">exclude</span> = [
<span class="s2">"window_type = 'dock'"</span>,
<span class="s2">"window_type = 'desktop'"</span>
];
<span class="c">#################################
#
# Fading
#
#################################
</span>
<span class="c"># Fade windows during opacity changes.
</span><span class="n">fading</span> = <span class="n">true</span>;
<span class="c"># The time between steps in a fade in milliseconds. (default 10).
</span><span class="n">fade</span>-<span class="n">delta</span> = <span class="m">4</span>;
<span class="c"># Opacity change between steps while fading in. (default 0.028).
</span><span class="n">fade</span>-<span class="n">in</span>-<span class="n">step</span> = <span class="m">0</span>.<span class="m">03</span>;
<span class="c"># Opacity change between steps while fading out. (default 0.03).
</span><span class="n">fade</span>-<span class="n">out</span>-<span class="n">step</span> = <span class="m">0</span>.<span class="m">03</span>;
<span class="c"># Fade windows in/out when opening/closing
# no-fading-openclose = true;
</span>
<span class="c"># Specify a list of conditions of windows that should not be faded.
</span><span class="n">fade</span>-<span class="n">exclude</span> = [ ];
<span class="c">#################################
#
# Other
#
#################################
</span>
<span class="c"># Try to detect WM windows and mark them as active.
</span><span class="n">mark</span>-<span class="n">wmwin</span>-<span class="n">focused</span> = <span class="n">true</span>;
<span class="c"># Mark all non-WM but override-redirect windows active (e.g. menus).
</span><span class="n">mark</span>-<span class="n">ovredir</span>-<span class="n">focused</span> = <span class="n">true</span>;
<span class="c"># Use EWMH _NET_WM_ACTIVE_WINDOW to determine which window is focused instead of using FocusIn/Out events.
# Usually more reliable but depends on a EWMH-compliant WM.
</span><span class="n">use</span>-<span class="n">ewmh</span>-<span class="n">active</span>-<span class="n">win</span> = <span class="n">true</span>;
<span class="c"># Detect rounded corners and treat them as rectangular when --shadow-ignore-shaped is on.
</span><span class="n">detect</span>-<span class="n">rounded</span>-<span class="n">corners</span> = <span class="n">true</span>;
<span class="c"># Detect _NET_WM_OPACITY on client windows, useful for window managers not passing _NET_WM_OPACITY of client windows to frame windows.
# This prevents opacity being ignored for some apps.
# For example without this enabled my xfce4-notifyd is 100% opacity no matter what.
</span><span class="n">detect</span>-<span class="n">client</span>-<span class="n">opacity</span> = <span class="n">true</span>;
<span class="c"># Specify refresh rate of the screen.
# If not specified or 0, compton will try detecting this with X RandR extension.
</span><span class="n">refresh</span>-<span class="n">rate</span> = <span class="m">0</span>;
<span class="c"># Set VSync method. VSync methods currently available:
# none: No VSync
# drm: VSync with DRM_IOCTL_WAIT_VBLANK. May only work on some drivers.
# opengl: Try to VSync with SGI_video_sync OpenGL extension. Only work on some drivers.
# opengl-oml: Try to VSync with OML_sync_control OpenGL extension. Only work on some drivers.
# opengl-swc: Try to VSync with SGI_swap_control OpenGL extension. Only work on some drivers. Works only with GLX backend. Known to be most effective on many drivers. Does not actually control paint timing, only buffer swap is affected, so it doesn't have the effect of --sw-opti unlike other methods. Experimental.
# opengl-mswc: Try to VSync with MESA_swap_control OpenGL extension. Basically the same as opengl-swc above, except the extension we use.
# (Note some VSync methods may not be enabled at compile time.)
</span><span class="n">vsync</span> = <span class="s2">"opengl-swc"</span>;
<span class="c"># Enable DBE painting mode, intended to use with VSync to (hopefully) eliminate tearing.
# Reported to have no effect, though.
</span><span class="n">dbe</span> = <span class="n">false</span>;
<span class="c"># Painting on X Composite overlay window. Recommended.
</span><span class="n">paint</span>-<span class="n">on</span>-<span class="n">overlay</span> = <span class="n">true</span>;
<span class="c"># Limit compton to repaint at most once every 1 / refresh_rate second to boost performance.
# This should not be used with --vsync drm/opengl/opengl-oml as they essentially does --sw-opti's job already,
# unless you wish to specify a lower refresh rate than the actual value.
</span><span class="n">sw</span>-<span class="n">opti</span> = <span class="n">false</span>;
<span class="c"># Unredirect all windows if a full-screen opaque window is detected, to maximize performance for full-screen windows, like games.
# Known to cause flickering when redirecting/unredirecting windows.
# paint-on-overlay may make the flickering less obvious.
</span><span class="n">unredir</span>-<span class="n">if</span>-<span class="n">possible</span> = <span class="n">true</span>;
<span class="c"># Specify a list of conditions of windows that should always be considered focused.
</span><span class="n">focus</span>-<span class="n">exclude</span> = [ ];
<span class="c"># Use WM_TRANSIENT_FOR to group windows, and consider windows in the same group focused at the same time.
</span><span class="n">detect</span>-<span class="n">transient</span> = <span class="n">true</span>;
<span class="c"># Use WM_CLIENT_LEADER to group windows, and consider windows in the same group focused at the same time.
# WM_TRANSIENT_FOR has higher priority if --detect-transient is enabled, too.
</span><span class="n">detect</span>-<span class="n">client</span>-<span class="n">leader</span> = <span class="n">true</span>;
<span class="c">#################################
#
# Window type settings
#
#################################
</span>
<span class="n">wintypes</span>:
{
<span class="n">tooltip</span> =
{
<span class="c"># fade: Fade the particular type of windows.
</span> <span class="n">fade</span> = <span class="n">true</span>;
<span class="c"># shadow: Give those windows shadow
</span> <span class="n">shadow</span> = <span class="n">false</span>;
<span class="c"># opacity: Default opacity for the type of windows.
</span> <span class="n">opacity</span> = <span class="m">0</span>.<span class="m">85</span>;
<span class="c"># focus: Whether to always consider windows of this type focused.
</span> <span class="n">focus</span> = <span class="n">true</span>;
};
};</code></pre></div>
<p>Details on what each of these options does can be found <a href="https://github.com/chjj/compton/blob/master/man/compton.1.asciidoc">here</a>. Some of them might need adjusting if you have crappy graphics drivers but should work for anyone with reasonable, up to date drivers <span class="amp">&</span> some kind of 3D graphics card.</p>
<p>It worked perfectly for me, on both my desktop dual monitor setup on an NVidia <span class="caps">8800GTS</span> using the current xorg-edgers driver, 313.30 <a class="footnote-ref" href="#_footnote_4" id="_footnoteref_4" role="doc-noteref" title="View footnote 4">[4]</a> - and also on my laptop with a some sort of crappy Mobility Radeon. By the look of the documentation, the most likely settings that might cause problems with drivers would be <code>vsync</code> and <code>backend</code>.</p></section>
<section class="doc-section level-1"><h2 id="_start_compton_for_the_current_session">Start Compton for the Current Session</h2><p>Now we’re going to make sure this is all working by starting compton. Press Alt+F2, type <code>compton</code> in the Application Launcher box, then press enter:</p>
<div class="image-block"><img alt="Screenshot of the XFCE Applications Filder launching Compton." src="https://duncanlock.net/images/posts/how-to-switch-to-compton-for-beautiful-tear-free-compositing-in-xfce/xfce-application-finder-launching-compton.png"/></div>
<p>Your screen will flicker and you should now have glassy smooth, tear free window dragging, with drop shadows and beautiful fading on window open/close <span class="amp">&</span> desktop switching, etc… Try dragging and few windows around, switching workspaces and open and closing things. Bathe in the smoothness.</p></section>
<section class="doc-section level-1"><h2 id="_set_compton_to_auto_start">Set Compton to auto-start</h2><p>Assuming that worked, we’ll make Compton start at startup. Go into the Applications menu and click ‘Settings Manager’, then click ‘Session and Startup’, then select the ‘Application Autostart’ tab:</p>
<figure class="image-block"><img alt="Screenshot of the XFCE Settings Manager - Session and Startup window" src="https://duncanlock.net/images/posts/how-to-switch-to-compton-for-beautiful-tear-free-compositing-in-xfce/xfce-settings-manager-session-and-startup-add-application.png" width="showing the filled in 'Add application' box."/>
<figcaption>Figure 1. Click the ‘Add’ button, then fill in the boxes like this.</figcaption></figure></section>
<section class="doc-section level-1"><h2 id="_excluding_some_windows_using_xwininfo_and_shadow_exclude">Excluding some windows using xwininfo and shadow-exclude</h2><figure class="image-block align-right"><img alt="xfce notify osd window corner" src="https://duncanlock.net/images/posts/how-to-switch-to-compton-for-beautiful-tear-free-compositing-in-xfce/xfce-notify-osd-window-corner.png"/>
<figcaption>Figure 2. Notice the square background behind the rounded corners on this volume notification.</figcaption></figure>
<p>You probably don’t want shadows on every window - they don’t work very well on notification popups, for example.</p>
<p>To exclude certain types of window, or certain applications, from having shadows, you can set the <code>shadow-exclude</code> setting. This setting is a list of conditions to match windows to. The simplest one is a wild card match on the window name, which is done something like this: <code>name *= 'Firefox'</code>.</p>
<p>Here’s an example from my config file. It excludes various notification popups, <span class="caps">VLC</span>, Chrome, Kupfer <a class="footnote-ref" href="#_footnote_5" id="_footnoteref_5" role="doc-noteref" title="View footnote 5">[5]</a> and other problem apps:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="conf"><span class="n">shadow</span>-<span class="n">exclude</span> = [
<span class="s2">"! name~=''"</span>,
<span class="s2">"name = 'Notification'"</span>,
<span class="s2">"name = 'Plank'"</span>,
<span class="s2">"name = 'Docky'"</span>,
<span class="s2">"name = 'Kupfer'"</span>,
<span class="s2">"name = 'xfce4-notifyd'"</span>,
<span class="s2">"name *= 'VLC'"</span>,
<span class="s2">"name *= 'compton'"</span>,
<span class="s2">"name *= 'Chromium'"</span>,
<span class="s2">"name *= 'Chrome'"</span>,
<span class="s2">"name *= 'Firefox'"</span>,
<span class="s2">"class_g = 'Conky'"</span>,
<span class="s2">"class_g = 'Kupfer'"</span>,
<span class="s2">"class_g = 'Synapse'"</span>,
<span class="s2">"class_g ?= 'Notify-osd'"</span>,
<span class="s2">"class_g ?= 'Cairo-dock'"</span>,
<span class="s2">"class_g ?= 'Xfce4-notifyd'"</span>,
<span class="s2">"class_g ?= 'Xfce4-power-manager'"</span>
];</code></pre></div>
<p>To add to this, you will need to know either the name or the class that X11 uses to refer to the window. There’s a handy utility called <code>xwininfo</code> that will tell you this. To use it, run this from a console window:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>xwininfo <span class="nt">-stats</span> <span class="nt">-wm</span></code></pre></div>
<p>Your mouse cursor will turn into a little cross-hair. Use this to click on the window you want to know about and <code>xwininfo</code> will print out some information about it. For example, clicking on an <span class="caps">XFCE</span> notification bubble will print something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">xwininfo: Window id: 0x9a00073 "xfce4-notifyd"
Absolute upper-left X: 1390
Absolute upper-left Y: 16
Relative upper-left X: 0
Relative upper-left Y: 0
Width: 274
Height: 76
Depth: 32
Visual: 0xec
Visual Class: TrueColor
Border width: 0
Class: InputOutput
Colormap: 0x9a00003 (not installed)
Bit Gravity State: NorthWestGravity
Window Gravity State: NorthWestGravity
Backing Store State: NotUseful
Save Under State: no
Map State: IsViewable
Override Redirect State: no
Corners: +1390+16 -2064+16 -2064-1060 +1390-1060
-geometry 274x76+1390+16
Window manager hints:
Client accepts input or input focus: No
Initial state is Normal State
Displayed on all desktops
Window type:
Notification
Window state:
Sticky
Skip Pager
Skip Taskbar
Above
Process id: 23420 on host duncan-desktop
Frame extents: 0, 0, 0, 0</code></pre></div>
<p>The window name is on the end of the first line (<code>xfce4-notifyd</code> in this case) and the class and type are further down. <a href="https://github.com/chjj/compton/blob/master/man/compton.1.asciidoc#format-of-conditions">Click here for more information about Compton conditionals</a>. You can use this information to add exclusions for these windows to your config.</p>
<p>All done. If you have any improvements on this setup, let me know in <a href="#_article_comments_section">the comments</a>.</p>
<hr/>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References</h3></section></section><section aria-label="Footnotes" class="footnotes" role="doc-endnotes"><hr/><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">Some window managers have Compositing built in and some don’t. <a href="http://en.wikipedia.org/wiki/Compositing_window_manager">See here for more info</a>. <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">Some of this information here came from this <a href="http://www.neowin.net/forum/topic/1148464-using-compton-for-tear-free-compositing-in-xfce/">great guide by ViperAFK on the NeoWin formus</a> and <a href="http://ubuntuforums.org/showthread.php?t=2144468">this one by screaminj3sus on ubuntuforums.org</a>, as suggested by Saravanan Kumar in the comments. <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_3" role="doc-endnote">Compton code is on <a href="https://github.com/chjj/compton">GitHub</a> and the <span class="caps">PPA</span> is on <a href="https://launchpad.net/~richardgv/+archive/compton">Launchpad</a>. <a class="footnote-backref" href="#_footnoteref_3" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_4" role="doc-endnote">xorg-edgers: “Packages for those who think development versions, experimental and unstable are for old ladies. We want our crack straight from upstream git! Well, straight, we want it built and packaged so we don’t need to know what we’re doing, except that we will break our X and put our computers on fire.” <a href="https://launchpad.net/~xorg-edgers">Use at your own risk!</a> <a class="footnote-backref" href="#_footnoteref_4" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_5" role="doc-endnote"><a href="https://live.gnome.org/Kupfer">Kupfer: An extremely lightweight quick launcher, like Gnome <span class="caps">DO</span></a>, “a convenient command and access tool”, is a program that can launch applications and open documents, and access different types of objects and act on them. <a class="footnote-backref" href="#_footnoteref_5" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>Magic Phone Numbers: My VOIP Setup, with voip.ms2013-05-31T14:46:45-07:002021-06-12T20:16:37-07:00Duncan Locktag:duncanlock.net,2013-05-31:/blog/2013/05/31/magic-phone-numbers-my-voip-setup-with-voipms/<figure class="image-block"><img src="https://duncanlock.net/images/posts/magic-phone-numbers-my-voip-setup-with-voipms/magic-number-diagram.png" alt="Schmatic diagram showing two phones" width="linked by a magic cloud" height="calling from the UK to Canada - via the cloud - for free.">
<figcaption>Figure 1. In an ideal world, I’d like something like this - and vice versa, please.</figcaption></figure>
<p>My wife and I currently live in Canada, and our families are back in the <span class="caps">UK</span>. We’d been using Skype to chat with people in the <span class="caps">UK</span>, but I was getting frustrated with Skype’s limitations: it either ties you to a computer and WiFi, or running a battery hungry app which slows down your phone. It doesn’t work very well over the 3G phone network here and while it’s free to make Skype to Skype calls, it’s fairly expensive calling real phones. I didn’t really want to buy a dedicated Skype phone - we both already have mobile/cell phones. And, to top it all off, the Skype software for Linux is the poor cousin of the Windows one …</p><figure class="image-block"><img src="https://duncanlock.net/images/posts/magic-phone-numbers-my-voip-setup-with-voipms/magic-number-diagram.png" alt="Schmatic diagram showing two phones" width="linked by a magic cloud" height="calling from the UK to Canada - via the cloud - for free.">
<figcaption>Figure 1. In an ideal world, I’d like something like this - and vice versa, please.</figcaption></figure>
<p>My wife and I currently live in Canada, and our families are back in the <span class="caps">UK</span>. We’d been using Skype to chat with people in the <span class="caps">UK</span>, but I was getting frustrated with Skype’s limitations: it either ties you to a computer and WiFi, or running a battery hungry app which slows down your phone. It doesn’t work very well over the 3G phone network here and while it’s free to make Skype to Skype calls, it’s fairly expensive calling real phones. I didn’t really want to buy a dedicated Skype phone - we both already have mobile/cell phones. And, to top it all off, the Skype software for Linux is the poor cousin of the Windows one, or pretty much unusable if you run 64-bit Linux.</p>
<p>In addition to these Skype specific issues, our parents both live out in the English countryside, and their internet connections are pretty slow and unreliable. It would be better if they didn’t have to use the internet on their end at all.</p>
<p>So, what I really wanted was something that I could easily use on my Android phone without having to run yet another app, and that was cheaper and easier to use. I also wanted something that could be used exactly like a regular local phone number – just call it and ring a phone on the other side of the world.</p>
<section class="doc-section level-1"><h2 id="_what_i_wanted_was_a_magic_phone_number">What I wanted was a Magic Phone Number</h2><p>After a lot of reading <span class="amp">&</span> research <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>, I decided that I wanted a <span class="caps">VOIP</span> (Voice Over <span class="caps">IP</span>) <a class="footnote-ref" id="_footnoteref_2" href="#_footnote_2" title="View footnote 2" role="doc-noteref">[2]</a> setup of some sort.</p>
<p>Unlike Skype, <span class="caps">VOIP</span> is an open standard, which means that anyone can create software and devices for it, not just one company. Amongst other things, this means that there are lots of <span class="caps">VOIP</span> soft-phones <a class="footnote-ref" id="_footnoteref_3" href="#_footnote_3" title="View footnote 3" role="doc-noteref">[3]</a> available for every platform and, even better, Android has built in support for making and receiving <span class="caps">VOIP</span> calls – so it would just work on our existing phones, without needing to install anything.</p>
<p>There are also lots of <span class="caps">VOIP</span> services available that provide lots of other features, like <span class="caps">DID</span> (Direct Inward Dialing - i.e. real phone numbers), voice-mail, call forwarding, ring groups, <span class="caps">IVR</span> (Interactive Voice Response - i.e. Press One for…​, Two for…​)` etc…​</p>
<p>I eventually chose <a href="http://voip.ms/">http://voip.ms/</a>, because of their extreme flexibility, <a href="http://wiki.voip.ms/article/Features">features</a> and very low wholesale-style costs.</p>
<p>Because it’s so flexible, it easily allowed us to get what we needed, along with plenty of bonus extras, for almost no cost.</p></section>
<section class="doc-section level-1"><h2 id="_getting_a_real_phone_number">Getting a Real Phone Number</h2><p>The first thing I wanted was a real <span class="caps">UK</span> phone number that people there could call, that would ring our phones in Canada. Ideally it would be a local number for our parents - familiar and free or cheap to call from a landline phone.</p>
<p>These are called <span class="caps">DID</span> (Direct Inward Dialing - i.e. real phone numbers) numbers - and you can get one for <a href="https://www.voip.ms/intldids.php">practically anywhere</a>. <span class="caps">DID</span>’s for many places have a choice of either pay as you go, or monthly flat rate which includes unlimited inbound calls to that number; the <span class="caps">UK</span> ones only offer flat rate: $0 setup fee and $4 per month unlimited inbound calls. You just choose the area code you want and click Order.</p>
<p>Congratulations, you now have a <span class="caps">UK</span> phone number! The process for getting a phone number practically anywhere else in the world is the same. Now you just need to decide what happens when someone calls it.</p>
<section class="doc-section level-2"><h3 id="_routing">Routing</h3><p>There are <em>lots</em> of options for routing your incoming calls: send them to voicemail, send them to a Digital Receptionist/<span class="caps">IVR</span> (Interactive Voice Response - i.e. Press One for…​, Two for…​), forward them to another number, play them a recording, etc…​</p>
<p>I wanted to give people the option of being put through to either me, or my wife, so I wanted to prompt them to choose: ‘Press 1 to call Duncan, Press 2 to…​’ – so I needed to send callers to an <span class="caps">IVR</span>:</p></section>
<section class="doc-section level-2"><h3 id="_interactive_voice_response_ivr">Interactive Voice Response (<span class="caps">IVR</span>)</h3><p>This is just a simple voice prompt system. It plays the caller a recording that you supply and then does things in response to them pressing different digits. You can route each option anywhere you can route a normal call. I set up 1 to call me, 2 to call my wife, 3 to leave me a voicemail and 4 to leave one for my wife:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/magic-phone-numbers-my-voip-setup-with-voipms/uk-voip-diagram.png" alt="Schematic blueprint style diagram showing a UK caller" width="calling a local UK landline" height="being routed to an IVR menu in the 'magic cloud'">
<figcaption>Figure 2. Simple <span class="caps">UK</span> <span class="caps">IVR</span> setup, works out to free calls, plus $4 per month flat rate.</figcaption></figure>
<p>Now I wanted the same thing, in reverse – a local number that I could call in Canada that would let me call the <span class="caps">UK</span>.</p></section></section>
<section class="doc-section level-1"><h2 id="_phone_to_phone_calling_via_the_cloud">Phone to Phone calling, via the Cloud</h2><p>So I purchased another <span class="caps">DID</span> number, a local Vancouver number, that we could call for free from our cell phones. I then created another <span class="caps">IVR</span>, so that when we called that number it would give us a menu of people in the <span class="caps">UK</span> to call:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/magic-phone-numbers-my-voip-setup-with-voipms/screenshot-13-05-31_03-30-35-pm.png" alt="screenshot 13 05 31 03 30 35 pm">
<figcaption>Figure 3. Building an <span class="caps">IVR</span> menu in voip.ms. On the right - the digit the caller pressed, on the left, the destination. As you can see, there are lots of different options on the left - these are all the different places you can send callers.</figcaption></figure>
<p>Because our cell phone plan gives us unlimited included/free talk time to local numbers, this means that we can call our local <span class="caps">DID</span> number for free and talk to people in the <span class="caps">UK</span> (or anywhere else), for as long as we like - no data connection required.</p>
<p>There are two different plans available for the Vancouver <span class="caps">DID</span> number: either $1.99 per month plus 0.0149¢ per minute, or $5.95 per month flat rate. The flat rate plan is cheaper if you’re using more than 400 minutes per month - we’re currently using less than that, so we’re on the $1.99 plan.</p></section>
<section class="doc-section level-1"><h2 id="_counting_the_cost">Counting the Cost</h2><p>Last month, we spent a total of <strong>14 hours, 24 minutes, 55 seconds</strong> calling people in the <span class="caps">UK</span> and spent a total of <strong>$12.59</strong>. That’s…​ <em>a lot</em> of talking, for not very much cash.</p>
<p>This gives us what we wanted - magic numbers that we can use to call anyone in the world from our cell phones - and that anyone can use to call us - at very low prices.</p></section>
<section class="doc-section level-1"><h2 id="_but_wait_theres_more">But wait, there’s more…​</h2><p>This just scratches the surface of the things you can do using <span class="caps">VOIP</span> <span class="amp">&</span> voip.ms. Here’s a taster of some of the other things that you can do, some of this we’re already using and some we might use in the future:</p>
<div class="dlist"><dl><dt><a href="http://wiki.voip.ms/article/Voicemail">Free Voicemail for anything</a></dt><dd>You can route any call to voicemail to take a message. The system can then email the recording (as a .wav file attachment) to any inbox.</dd><dt><a href="http://wiki.voip.ms/article/Caller_ID">CallerID</a></dt><dd>You can pass-through the CallerID from your phone when you use <span class="caps">DID</span> numbers to make outbound calls if you want, or you can set them yourself. You can also have the system report CallerID’s on incoming calls, so that your phone will tell you who’s calling.</dd><dt><span class="caps">VOIP</span> to <span class="caps">VOIP</span> calling, <span class="caps">VOIP</span> to Phone calling (aka free long distance)</dt><dd>You don’t have to use the <span class="caps">POTS</span> (Plain Old Telephone System) at all - you can make pure data calls over the internet, either to regular phones or <span class="caps">SIP</span>/<span class="caps">VOIP</span> numbers. This means that you can use WiFi to make calls without using your cell phone minutes at all, or indeed having a <span class="caps">SIM</span> card or a phone - you can use a softphone on any computer or laptop to make calls.</dd><dt><a href="http://wiki.voip.ms/article/SMS"><span class="caps">SMS</span> text messaging</a></dt><dd>I haven’t figured this out yet, but you can send <span class="amp">&</span> receive <span class="caps">SMS</span> text messages, using your <span class="caps">DID</span> numbers. This is a new feature currently <span class="caps">US</span> only (and free) - will be 1¢ per text from 2014.</dd><dt><a href="http://wiki.voip.ms/article/DISA"><span class="caps">DISA</span> - Direct Inward System Access</a></dt><dd>This allows you to make outgoing calls, to anyone, with no setup. You just dial to your <span class="caps">DID</span> number, provide a 4 digit <span class="caps">PIN</span>, then you can dial out to any number in the world, using <a href="http://www.voip.ms/rates.php">voip.ms’s cheap termination rates</a>.</dd><dt><a href="http://wiki.voip.ms/article/Callback">Callback</a></dt><dd>You can define a number to be called back by voip.ms, in order to receive a dial tone and place outgoing calls. This could be useful if you want to place a call and you are not at home or don’t have access to your voip device: you call the number, hang up and it calls the predefined number. You pick up and you get a dial tone - and you can then dial any phone number.</dd><dt><a href="http://wiki.voip.ms/article/CallerID_Filtering">CallerID Filtering</a></dt><dd>Allows you to filter the incoming calls to your <span class="caps">DID</span> numbers that came from specific numbers, area code or even anonymous numbers. For example, if you receive annoying incoming calls from a telemarketing company you can create a filter to route all the calls to a recording that plays the message “That number is no longer in service, please hang up and try again”, amongst several other options. You can also flip this around and filter out everyone except certain numbers, creating a private line that’s impervious to telemarketers.</dd></dl></div>
<p>There are also loads of ‘professional’ type features designed for big offices - calling cues, ring groups, failover, time conditions, etc…​ Lots more details here: <a href="http://wiki.voip.ms/article/Features">http://wiki.voip.ms/article/Features</a>.</p>
<p>If you’re sold on voip, here’s the <a href="http://wiki.voip.ms/article/Getting_Started">voip.ms getting started guide</a> – and if you’ve got any questions, please just ask in the comments!</p>
<hr>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References:</h3></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">Thanks very much to Jay Parlar, who wrote up his voip setup <a href="http://parlar.ca/blog/2011/8/8/my-voip-setup-with-voipms.html">here</a>. <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">What is <span class="caps">VOIP</span>: <a href="http://en.wikipedia.org/wiki/Voice_over_IP">http://en.wikipedia.org/wiki/Voice_over_IP</a> <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_3" role="doc-endnote">A soft-phone is a piece of software, like Skype, that allows you to make phone calls on a computer. Unlike Skype, most of them support <span class="caps">SIP</span>, <span class="caps">STUN</span> and <span class="caps">VOIP</span>. <a class="footnote-backref" href="#_footnoteref_3" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>Better Figures & Images Plugin for Pelican2013-05-29T15:49:49-07:002013-05-29T15:49:49-07:00Duncan Locktag:duncanlock.net,2013-05-29:/blog/2013/05/29/better-figures-images-plugin-for-pelican/<section class="doc-section level-1"><h2 id="_how_figures_images_work_in_pelican_by_default">How Figures <span class="amp">&</span> Images work in Pelican, by default</h2><p>By default Pelican does a great job with figures and images, thanks to built-in support in ReStructuredText <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>[1] . Pelican will turn this rst input:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">.. figure:: {static}/images/better-figures-images-plugin-for-pelican/dummy-200x200.png
:align: right
This is the caption of the figure.
The legend consists of all elements after the caption. In this case, the legend consists of this paragraph.</code></pre></div>
<p>into this <span class="caps">HTML</span> output:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><div</span> <span class="na">class=</span><span class="s">"figure align-right"</span><span class="nt">></span>
<span class="nt"><img</span> <span class="na">alt=</span><span class="s">"/static/images/dummy-200x200.png"</span> <span class="na">src=</span><span class="s">"/static/images/dummy-200x200.png"</span> <span class="nt">/></span>
<span class="nt"><p</span> <span class="na">class=</span><span class="s">"caption"</span><span class="nt">></span>This is the caption of the figure.<span class="nt"></p></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"legend"</span><span class="nt">></span>The legend consists of all elements after the caption. In this case, the legend consists of this paragraph<span class="nt"></div></span>
<span class="nt"></div></span></code></pre></div>
<p>Which, given this <span class="caps">CSS</span>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Styles for Figures & Images */</span>
<span class="nt">img</span> <span class="p">{</span>
<span class="nl">border</span><span class="p">:</span> <span class="m">1px</span> <span class="nb">solid</span> <span class="m">#bbb</span><span class="p">;</span>
<span class="nl">border-radius</span><span class="p">:</span> <span class="m">3px</span><span class="p">;</span>
<span class="nl">padding</span><span class="p">:</span> <span class="m">4px</span><span class="p">;</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">left</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">1em …</span></code></pre></div></section><section class="doc-section level-1"><h2 id="_how_figures_images_work_in_pelican_by_default">How Figures <span class="amp">&</span> Images work in Pelican, by default</h2><p>By default Pelican does a great job with figures and images, thanks to built-in support in ReStructuredText <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>[1] . Pelican will turn this rst input:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">.. figure:: {static}/images/better-figures-images-plugin-for-pelican/dummy-200x200.png
:align: right
This is the caption of the figure.
The legend consists of all elements after the caption. In this case, the legend consists of this paragraph.</code></pre></div>
<p>into this <span class="caps">HTML</span> output:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><div</span> <span class="na">class=</span><span class="s">"figure align-right"</span><span class="nt">></span>
<span class="nt"><img</span> <span class="na">alt=</span><span class="s">"/static/images/dummy-200x200.png"</span> <span class="na">src=</span><span class="s">"/static/images/dummy-200x200.png"</span> <span class="nt">/></span>
<span class="nt"><p</span> <span class="na">class=</span><span class="s">"caption"</span><span class="nt">></span>This is the caption of the figure.<span class="nt"></p></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"legend"</span><span class="nt">></span>The legend consists of all elements after the caption. In this case, the legend consists of this paragraph<span class="nt"></div></span>
<span class="nt"></div></span></code></pre></div>
<p>Which, given this <span class="caps">CSS</span>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="c">/* Styles for Figures & Images */</span>
<span class="nt">img</span> <span class="p">{</span>
<span class="nl">border</span><span class="p">:</span> <span class="m">1px</span> <span class="nb">solid</span> <span class="m">#bbb</span><span class="p">;</span>
<span class="nl">border-radius</span><span class="p">:</span> <span class="m">3px</span><span class="p">;</span>
<span class="nl">padding</span><span class="p">:</span> <span class="m">4px</span><span class="p">;</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">left</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">1em</span> <span class="m">1em</span> <span class="m">1em</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">box-shadow</span><span class="p">:</span> <span class="m">4px</span> <span class="m">4px</span> <span class="m">4px</span> <span class="m">0px</span> <span class="n">rgba</span><span class="p">(</span><span class="m">128</span><span class="p">,</span> <span class="m">128</span><span class="p">,</span> <span class="m">128</span><span class="p">,</span> <span class="m">0.5</span><span class="p">);</span>
<span class="p">}</span>
<span class="nt">img</span><span class="nc">.align-right</span> <span class="p">{</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">right</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">1em</span> <span class="m">0</span> <span class="m">1em</span> <span class="m">1em</span><span class="p">;</span>
<span class="p">}</span>
<span class="nc">.figure</span> <span class="p">{</span>
<span class="nl">border</span><span class="p">:</span> <span class="m">1px</span> <span class="nb">solid</span> <span class="m">#bbb</span><span class="p">;</span>
<span class="nl">border-radius</span><span class="p">:</span> <span class="m">3px</span><span class="p">;</span>
<span class="nl">padding</span><span class="p">:</span> <span class="m">4px</span><span class="p">;</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">left</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">0</span> <span class="m">1em</span> <span class="m">1em</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">box-shadow</span><span class="p">:</span> <span class="m">4px</span> <span class="m">4px</span> <span class="m">4px</span> <span class="m">0px</span> <span class="n">rgba</span><span class="p">(</span><span class="m">128</span><span class="p">,</span> <span class="m">128</span><span class="p">,</span> <span class="m">128</span><span class="p">,</span> <span class="m">0.5</span><span class="p">);</span>
<span class="p">}</span>
<span class="nc">.figure.align-right</span> <span class="p">{</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">right</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">0</span> <span class="m">0</span> <span class="m">1em</span> <span class="m">1em</span><span class="p">;</span>
<span class="p">}</span>
<span class="nc">.figure.align-left</span> <span class="p">{</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">left</span><span class="p">;</span>
<span class="p">}</span>
<span class="nc">.figure</span> <span class="nt">img</span> <span class="p">{</span>
<span class="nl">border</span><span class="p">:</span> <span class="nb">none</span><span class="p">;</span>
<span class="nl">border-radius</span><span class="p">:</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">padding</span><span class="p">:</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">box-shadow</span><span class="p">:</span> <span class="nb">none</span><span class="p">;</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">left</span><span class="p">;</span>
<span class="p">}</span>
<span class="nc">.figure</span> <span class="nt">p</span><span class="nc">.caption</span> <span class="p">{</span>
<span class="nl">margin</span><span class="p">:</span> <span class="m">0</span><span class="p">;</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">80%</span><span class="p">;</span>
<span class="nl">padding</span><span class="p">:</span> <span class="m">4px</span><span class="p">;</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">left</span><span class="p">;</span>
<span class="nl">text-align</span><span class="p">:</span> <span class="nb">left</span><span class="p">;</span>
<span class="p">}</span>
<span class="nc">.figure</span> <span class="nt">div</span><span class="nc">.legend</span> <span class="p">{</span>
<span class="nl">padding</span><span class="p">:</span> <span class="m">4px</span><span class="p">;</span>
<span class="nl">font-size</span><span class="p">:</span> <span class="m">80%</span><span class="p">;</span>
<span class="nl">color</span><span class="p">:</span> <span class="m">#aaa</span><span class="p">;</span>
<span class="nl">font-style</span><span class="p">:</span> <span class="nb">italic</span><span class="p">;</span>
<span class="nl">float</span><span class="p">:</span> <span class="nb">left</span><span class="p">;</span>
<span class="p">}</span></code></pre></div>
<p>…​will look something like this:</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/screenshot-13-04-29_16-42-00-pm.png" alt="screenshot 13 04 29 16 42 00 pm"></div>
<p>This is great, but it’s not <em>quite</em> what I wanted. I wanted the caption under the image and then the figure to shrink to fit the size of the image it contains. It turns out that this is impossible in <span class="caps">HTML</span> <span class="amp">&</span> <span class="caps">CSS</span> unless you give the browser an explicit <code>width</code> attribute for the <code>img</code> and the containing <code>div</code>.</p>
<p>If you actually do that, and add a <code>style="width: 200px; height: auto;"</code> attribute to both the <code>div</code> and the <code>img</code>, you get this - which <em>is</em> what I wanted:</p>
<section class="paragraph"><h6 class="block-title">This is the caption of the figure. The legend consists of all elements after the caption. In this</h6><p class="right">case, the legend consists of this paragraph. image::{static}/images/posts/better-figures-images-plugin-for-pelican/dummy-200x200.png[]</p></section>
<p>The problem with this, is that it means that you need to supply a width attribute containing the actual pixel width of the image, for each and every image you use. This would be <em>very</em> tedious to do by hand, so I wrote a Pelican plugin to do this, plus a couple of other related things, for me.</p></section>
<section class="doc-section level-1"><h2 id="_what_the_better_figures_images_plugin_does">What the Better Figures <span class="amp">&</span> Images plugin does</h2><div class="ulist"><ul><li>Adds a <code>style="width: ???px; height: auto;"</code> attribute to any <code><img></code> tags in the content, by automatically checking the dimensions of the actual image file on disk and adding the appropriate attribute to the <code><img></code> tag.</li><li>Also finds any <code><div class="figures"></code> tags in the content which contain images - and adds the same style attribute to them.</li><li>If the <code>RESPONSIVE_IMAGES</code> setting is true, it adds <code>style="width: ???px; max-width: 100%; height: auto;"</code> instead.</li><li>Corrects Alt text: If an img alt attribute = the image filename, it sets it to “”</li><li>Inserts automatic figure numbers into figure captions, if <code>FIGURE_NUMBERS == True</code> in global config, or figure_numbers exists in article metadata.</li></ul></div>
<p>Assuming that the image is 250px wide, it turns output like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><div</span> <span class="na">class=</span><span class="s">"figure"</span><span class="nt">></span>
<span class="nt"><img</span> <span class="na">alt=</span><span class="s">"/static/images/image.jpg"</span> <span class="na">src=</span><span class="s">"/static/images/better-figures-images-plugin-for-pelican/image.jpg"</span> <span class="nt">/></span>
<span class="nt"><p</span> <span class="na">class=</span><span class="s">"caption"</span><span class="nt">></span>
This is the caption of the figure.
<span class="nt"></p></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"legend"</span><span class="nt">></span>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
<span class="nt"></div></span>
<span class="nt"></div></span></code></pre></div>
<p>into output like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><div</span> <span class="na">class=</span><span class="s">"figure"</span> <span class="na">style=</span><span class="s">"width: 250px; height: auto;"</span><span class="nt">></span>
<span class="nt"><img</span> <span class="na">style=</span><span class="s">"width: 250px; height: auto;"</span> <span class="na">alt=</span><span class="s">""</span> <span class="na">src=</span><span class="s">"/static/images/image.jpg"</span> <span class="nt">/></span>
<span class="nt"><p</span> <span class="na">class=</span><span class="s">"caption"</span><span class="nt">></span>
This is the caption of the figure.
<span class="nt"></p></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"legend"</span><span class="nt">></span>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
<span class="nt"></div></span>
<span class="nt"></div></span></code></pre></div>
<p>or this, if <code>RESPONSIVE_IMAGES = True</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><div</span> <span class="na">class=</span><span class="s">"figure"</span> <span class="na">style=</span><span class="s">"width: 250px; max-width: 100%; height: auto;"</span><span class="nt">></span>
<span class="nt"><img</span> <span class="na">style=</span><span class="s">"width: 250px; max-width: 100%; height: auto;"</span> <span class="na">alt=</span><span class="s">""</span> <span class="na">src=</span><span class="s">"/static/images/image.jpg"</span> <span class="nt">/></span>
<span class="nt"><p</span> <span class="na">class=</span><span class="s">"caption"</span><span class="nt">></span>
This is the caption of the figure.
<span class="nt"></p></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"legend"</span><span class="nt">></span>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
<span class="nt"></div></span>
<span class="nt"></div></span></code></pre></div>
<p>or this, if <code>FIGURE_NUMBERS</code> is also True:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><div</span> <span class="na">class=</span><span class="s">"figure"</span> <span class="na">style=</span><span class="s">"width: 250px; max-width: 100%; height: auto;"</span><span class="nt">></span>
<span class="nt"><img</span> <span class="na">style=</span><span class="s">"width: 250px; max-width: 100%; height: auto;"</span> <span class="na">alt=</span><span class="s">""</span> <span class="na">src=</span><span class="s">"/static/images/image.jpg"</span> <span class="nt">/></span>
<span class="nt"><p</span> <span class="na">class=</span><span class="s">"caption"</span><span class="nt">></span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">"fig_num"</span> <span class="na">id=</span><span class="s">"fig_1"</span><span class="nt">></span>Figure 1: <span class="nt"></span></span>This is the caption of the figure.
<span class="nt"></p></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"legend"</span><span class="nt">></span>
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.
<span class="nt"></div></span>
<span class="nt"></div></span></code></pre></div></section>
<section class="doc-section level-1"><h2 id="_how_to_use_the_plugin">How to use the Plugin</h2><p>This plugin is now upstream in the main pelican-plugins repository, you can check that out like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>git clone git@github.com:getpelican/pelican-plugins.git</code></pre></div>
<p>It requires BeautifulSoup and <span class="caps">PIL</span>/Pillow - see the readme for details on installing these.</p>
<p>Then add something like this to your pelican config:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="c1"># Where to look for plugins
</span><span class="n">PLUGIN_PATH</span> <span class="o">=</span> <span class="s">'../pelican-plugins'</span>
<span class="c1"># Which plugins to enable
</span><span class="n">PLUGINS</span> <span class="o">=</span> <span class="p">[</span><span class="s">'better_figures_and_images'</span><span class="p">]</span></code></pre></div>
<p>Optionally, enable the responsive stuff via the plugin, by adding this to your config:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="c1"># Setting for the better_figures_and_images plugin
</span><span class="n">RESPONSIVE_IMAGES</span> <span class="o">=</span> <span class="bp">True</span></code></pre></div>
<p>Or add something like this to your theme’s <span class="caps">CSS</span>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nt">img</span><span class="o">,</span> <span class="nt">div</span><span class="nc">.figure</span> <span class="p">{</span> <span class="nl">max-width</span><span class="p">:</span> <span class="m">100%</span><span class="p">;</span> <span class="nl">height</span><span class="p">:</span> <span class="nb">auto</span><span class="p">;</span> <span class="p">}</span></code></pre></div>
<p>You can enable automatic figure numbering, by adding this to your config:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="python"><span class="c1"># Setting for the better_figures_and_images plugin
</span><span class="n">FIGURE_NUMBERS</span> <span class="o">=</span> <span class="bp">True</span></code></pre></div>
<p>or, to enable this on a per post basis, add this into the posts metadata:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="rst">:figure_numbers: True</code></pre></div>
<p>And that’s it - you should now have Better Figures <span class="amp">&</span> Images.</p>
<aside class="admonition-block note" role="note"><h6 class="block-title label-only"><span class="title-label">Note: </span></h6><p>Automatic Figure numbering is new and isn’t upstream yet - check out the <code>figure_numbers</code> branch from my git repo, <a href="https://github.com/dflock/pelican-plugins/tree/figure_numbers">here</a> if you want to use it.</p></aside></section>
<section class="doc-section level-1"><h2 id="_not_very_frequently_asked_questions">(Not Very) Frequently Asked Questions</h2><section class="doc-section level-2"><h3 id="_what_is_the_responsive_images_setting_for">What is the RESPONSIVE_IMAGES setting for?</h3><p>This site uses a responsive layout - it changes its layout and column widths based on the size of the screen or window you use to view it. This means, ideally, that any images contained inside those columns would also shrink or expand to fit, when the column they’re in changes. If they don’t, the images will break out of the columns if the column becomes too narrow.</p>
<p>The simplest way to do this, would be to add something like this to your <span class="caps">CSS</span>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="css"><span class="nt">img</span><span class="o">,</span> <span class="nt">div</span><span class="nc">.figure</span> <span class="p">{</span> <span class="nl">max-width</span><span class="p">:</span> <span class="m">100%</span><span class="p">;</span> <span class="nl">height</span><span class="p">:</span> <span class="nb">auto</span><span class="p">;</span> <span class="p">}</span></code></pre></div>
<p>This tells the browser that images can only ever be as wide as their container - i.e. 100% of the width of their parent element. This means that when the column that the image is in shrinks - and becomes smaller than the images native width - the image will be shrunk to fit inside.</p>
<p>Note that this isn’t the perfect solution and isn’t fully responsive - because there <a href="http://css-tricks.com/which-responsive-images-solution-should-you-use/">isn’t a perfect solution at the moment</a> - this provides a simple solution that gets me 80% of what I wanted: shrink to fit images that expand up to their full width (but no further) and stay inside their containers.</p></section>
<section class="doc-section level-2"><h3 id="_couldnt_you_just">Couldn’t you just…​</h3><p>Yes, you could just add that to your <span class="caps">CSS</span> and only have the plugin add the <code>width: ???px</code> part - this would work fine. If you want to do that, either don’t set <code>RESPONSIVE_IMAGES</code> in your pelican config, or set it to <code>False</code>.</p></section>
<section class="doc-section level-2"><h3 id="_so_why_is_there_a_responsive_images_setting_at_all">So why is there a RESPONSIVE_IMAGES setting at all?</h3><p>Um…​ It’s partially just there because this is the way I wrote the plugin initially, before I thought it through properly.</p>
<p>The reason I <em>left it in</em>, is twofold:</p>
<div class="olist arabic"><ol class="arabic"><li>It means that you can get responsive images and figures just by using this plugin - no need to mess with your theme’s <span class="caps">CSS</span> if you don’t want to.</li><li>Because there are lots of <a href="http://css-tricks.com/which-responsive-images-solution-should-you-use/">other ways</a> to fudge responsive images and I may decide to use one of the alternatives - and at some point, presumably an official standard way to do it will arrive. So I may want to do extra processing, add extra markup, or do other things to support future responsive image techniques here, so I left that hook in so that I could easily add it.</li></ol></div></section>
<section class="doc-section level-2"><h3 id="_why_are_you_messing_with_the_alt_text">Why are you messing with the <span class="caps">ALT</span> text?</h3><p>By default Pelican adds a default <code>alt</code> attribute to images that don’t have them - and sets it to the image’s filename.</p>
<p>This is well meaning, but wrong.</p>
<p>The <code>alt</code> attribute is meant to provide a textual alternative to the image, for people who can’t see the image, for some reason - they might be blind, using a screen reader, they might be using a text-only browser, they might be a search engine, the image might not have loaded for some reason, etc…​</p>
<p>Imagine that you are reading your page to someone over the phone. What would be the appropriate thing to do when you reach the image? <a class="footnote-ref" id="_footnoteref_2" href="#_footnote_2" title="View footnote 2" role="doc-noteref">[2]</a> What would you say about that image if you were describing the page over the phone to someone?</p>
<p>If you wouldn’t mention the image at all, then explicitly set the <code>alt</code> attribute to an empty string:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="html"><span class="nt"><img</span> <span class="na">alt=</span><span class="s">""</span> <span class="na">src=</span><span class="s">""</span> <span class="err">...</span> <span class="nt">/></span></code></pre></div>
<p>Otherwise, set it to whatever you would have said over the phone.</p>
<p>Why not just leave it out? Because screen readers tend to read the filename for images that don’t have an <code>alt</code> attribute. This also means that you <em>never</em> need to set the <code>alt</code> attribute to the image filename - that’s already there in the <code>src</code> attribute, if needed.</p></section></section>
<section class="doc-section level-1"><h2 id="_examples">Examples</h2><p>Here are a few working examples, showing the results of using the plugin. The original rst source for these are available in the plugins <code>/test</code> folder:</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-800x300.png" alt="dummy 800x300">
<figcaption>Figure 1. This image is wider than the column it’s in - try resizing the browser window. Because of the max-width: 100%, the image is resized to fit the column.</figcaption></figure>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-200x200.png" alt="A dummy placeholder image, 200x200 pixels square.">
<figcaption>Figure 2. This image is only 200px wide - smaller that the column it’s in. The max-width: 100% doesn’t stretch the image, because it’s also got a width: 200px - making it shrink to fit.</figcaption></figure>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur.</p>
<figure class="image-block align-right"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-250x300.png" alt="map to buried treasure 2">
<figcaption>Figure 3. This is the third image caption. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.</figcaption></figure>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/better-figures-images-plugin-for-pelican/dummy-200x200.png" alt="dummy 200x200"></div>
<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod
tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,
quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse
cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non
proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References:</h3></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">The two image directives: “image” and “figure” in reStructuredText: <a href="#images"><a class="bare" href="http://docutils.sourceforge.net/docs/ref/rst/directives.html#images">http://docutils.sourceforge.net/docs/ref/rst/directives.html#images</a></a> <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">Guidelines on <span class="caps">ALT</span> texts in <span class="caps">IMG</span> elements: <a href="http://www.cs.tut.fi/~jkorpela/html/alt.html">http://www.cs.tut.fi/~jkorpela/html/alt.html</a> <a href="http://www.456bereastreet.com/archive/200412/the_alt_and_title_attributes/">http://www.456bereastreet.com/archive/200412/the_alt_and_title_attributes/</a> <a href="http://diveintoaccessibility.info/day_21_ignoring_spacer_images.html">http://diveintoaccessibility.info/day_21_ignoring_spacer_images.html</a> <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>How I built this website, using Pelican: Part 1 - Setup2013-05-17T10:08:27-07:002021-06-12T20:39:33-07:00Duncan Locktag:duncanlock.net,2013-05-17:/blog/2013/05/17/how-i-built-this-website-using-pelican-part-1-setup/<p>As I <a href="https://duncanlock.net/blog/2013/04/26/welcome-to-the-new-site-same-as-the-old-site/">mentioned previously</a>, this site was put together using <a href="http://getpelican.com/">Pelican</a> - a static site generator, written in Python.</p>
<figure class="image-block"><a class="image" href="http://en.wikipedia.org/wiki/Brown_Pelican"><img src="https://duncanlock.net/images/posts/how-i-built-this-website-using-pelican-part-1-setup/pelecanus-occidentalis-diagram.png" alt="Blueprint style diagram showing a brown Pelican, flying. The diagram point out it's Yellow Head, Large beak and pouch for fishing, long neck, white chest and grey body."></a>
<figcaption>Figure 1. Pelecanus Occidentalis - the Brown Pelican. Original clipart Flying Pelican from OpenClipart, by molumen, Public Domain. More on Pelican, the bird.</figcaption></figure>
<p>Static site generators take your content, pour it into your templates and output the result as static pre-generated <span class="caps">HTML</span>, <span class="caps">CSS</span>, <span class="caps">JS</span> <span class="amp">&</span> image files. You can then just upload the resulting folder of output to your server and you’re done. All you need on the server is a web server of some sort, like Apache or Nginx - anything really - all it’s doing is serving static pages.</p>
<section class="doc-section level-1"><h2 id="_the_huge_advantage_of_this_setup_is_simplicity">The huge advantage of this setup is simplicity:</h2><div class="olist arabic"><ol class="arabic"><li>You can write your content in Markdown <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>, reStructuredText <a class="footnote-ref" id="_footnoteref_2" href="#_footnote_2" title="View footnote 2" role="doc-noteref">[2]</a> or AsciiDoc <a class="footnote-ref" id="_footnoteref_3" href="#_footnote_3" title="View footnote 3" role="doc-noteref">[3]</a> - all simple text formats, designed to facilitate writing and get out of …</li></ol></div></section><p>As I <a href="https://duncanlock.net/blog/2013/04/26/welcome-to-the-new-site-same-as-the-old-site/">mentioned previously</a>, this site was put together using <a href="http://getpelican.com/">Pelican</a> - a static site generator, written in Python.</p>
<figure class="image-block"><a class="image" href="http://en.wikipedia.org/wiki/Brown_Pelican"><img src="https://duncanlock.net/images/posts/how-i-built-this-website-using-pelican-part-1-setup/pelecanus-occidentalis-diagram.png" alt="Blueprint style diagram showing a brown Pelican, flying. The diagram point out it's Yellow Head, Large beak and pouch for fishing, long neck, white chest and grey body."></a>
<figcaption>Figure 1. Pelecanus Occidentalis - the Brown Pelican. Original clipart Flying Pelican from OpenClipart, by molumen, Public Domain. More on Pelican, the bird.</figcaption></figure>
<p>Static site generators take your content, pour it into your templates and output the result as static pre-generated <span class="caps">HTML</span>, <span class="caps">CSS</span>, <span class="caps">JS</span> <span class="amp">&</span> image files. You can then just upload the resulting folder of output to your server and you’re done. All you need on the server is a web server of some sort, like Apache or Nginx - anything really - all it’s doing is serving static pages.</p>
<section class="doc-section level-1"><h2 id="_the_huge_advantage_of_this_setup_is_simplicity">The huge advantage of this setup is simplicity:</h2><div class="olist arabic"><ol class="arabic"><li>You can write your content in Markdown <a class="footnote-ref" id="_footnoteref_1" href="#_footnote_1" title="View footnote 1" role="doc-noteref">[1]</a>, reStructuredText <a class="footnote-ref" id="_footnoteref_2" href="#_footnote_2" title="View footnote 2" role="doc-noteref">[2]</a> or AsciiDoc <a class="footnote-ref" id="_footnoteref_3" href="#_footnote_3" title="View footnote 3" role="doc-noteref">[3]</a> - all simple text formats, designed to facilitate writing and get out of your way. You can use whichever writing tool you prefer <a class="footnote-ref" id="_footnoteref_4" href="#_footnote_4" title="View footnote 4" role="doc-noteref">[4]</a>, as long as it can output plain text files.</li><li>Whenever you make changes, Pelican can automatically regenerate the site, so you can see your changes immediately.</li><li>When you’re done, Pelican can automatically upload the site to your web server, or you can do it, just by uploading a folder.</li><li>The web server generally requires no setup - all you need is a web server that can serve static content (which is all of them) - no extra software or configuration; no <span class="caps">PHP</span>, no database, no nothing - much less to go wrong.</li><li>Because you’ve only got one thing running on the server, you have much less exposure to security problems - no WordPress, no <span class="caps">PHP</span> - just the <span class="caps">OS</span> <span class="amp">&</span> the web server.</li><li>Because the server is only serving pre-generated static content, a Pelican site is very lightweight, using very few server resources.</li></ol></div>
<p><strong><span class="caps">NB</span></strong>: This tutorial is quite long. I go into detail, explain things and try to take you from zero to a complete, fully functional website, built the way a professional web developer would do it. If you just want a really quick Pelican jump start, try here: <a href="http://docs.getpelican.com/en/latest/getting_started.html">http://docs.getpelican.com/en/latest/getting_started.html</a></p></section>
<section class="doc-section level-1"><h2 id="_installation_basic_setup">Installation <span class="amp">&</span> Basic Setup</h2><p>I’m using Linux (Xubuntu), but doing this on Windows or Mac is quite similar - you’ll need the same things installed, but the details of installing them will be different. Installing <code>python-dev</code> and python packages that want to build C extensions on Windows…​ won’t work - I suggest you give <a href="http://code.activestate.com/pypm/">ActiveState’s Binary Package</a> manager a try if you’re on Windows.</p>
<p>Where I use <code>duncanlock</code> in the examples below, you should use your site name instead.</p>
<section class="doc-section level-2"><h3 id="_install">Install</h3><p>Pelican uses Python, so you’ll need that installed. If you’re on Mac or Linux, you’ll already have this, but you’ll probably need to <a href="http://www.activestate.com/activepython/downloads">install it on Windows</a>. Pelican supports Python 2.7.x or 3.3+, so use whichever you prefer. I’m doing this using Python 2.7.3, but I think everything should be the same on 3.x, although I haven’t tried it.</p>
<p>First, I’m going to install <code>pip</code> <a class="footnote-ref" id="_footnoteref_5" href="#_footnote_5" title="View footnote 5" role="doc-noteref">[5]</a>, <code>virtualenv</code> <a class="footnote-ref" id="_footnoteref_6" href="#_footnote_6" title="View footnote 6" role="doc-noteref">[6]</a> and <code>virtualenvwrapper</code> <a class="footnote-ref" id="_footnoteref_7" href="#_footnote_7" title="View footnote 7" role="doc-noteref">[7]</a>. These tools make working on python projects <em>much, much</em> easier. Later, we’re going to install some python packages that will attempt to build their C extensions during install, so we also need <code>python-dev</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>python-pip python-virtualenv virtualenvwrapper
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>python-dev</code></pre></div>
<p>Next, I’m going to tell <code>virtualenvwrapper</code> where I want it to put stuff, by adding this to my <code>~/.bashrc</code> file:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="bash"><span class="c"># virtualenvwrapper config</span>
<span class="nb">export </span><span class="nv">PROJECT_HOME</span><span class="o">=</span>~/dev
<span class="nb">export </span><span class="nv">WORKON_HOME</span><span class="o">=</span>~/dev/virtualenvs</code></pre></div>
<p>Now close and re-open your terminal. This will trigger a one-time setup for virtualenvwrapper. Then run:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>mkproject duncanlock.net-pelican</code></pre></div>
<p>which should do something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">New python executable in duncanlock.net-pelican/bin/python
Installing distribute.........done.
Installing pip...............done.
virtualenvwrapper.user_scripts creating
[...]
Creating /home/duncan/dev/duncanlock.net-pelican
Setting project for duncanlock.net-pelican to /home/duncan/dev/duncanlock.net-pelican</code></pre></div>
<p>You will now have a self-contained python virtual environment installed in <code>~/dev/virtualenvs/duncanlock.net-pelican</code> and a new folder in <code>~/dev/duncanlock.net-pelican</code>, to put your project files in. Your command prompt will change while this virtualenv is active - gaining a <code>(duncanlock.net-pelican)</code> at the beginning, so you know which virtualenv you’re in.</p>
<p>Next, we’re going to install Pelican and it’s dependencies into our virtual environment:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pip <span class="nb">install </span>pelican</code></pre></div>
<p>This should install the following things for you:</p>
<div class="dlist"><dl><dt>feedgenerator</dt><dd>to generate the Atom feeds</dd><dt>jinja2</dt><dd>for templating support</dd><dt>pygments</dt><dd>for syntax highlighting</dd><dt>docutils</dt><dd>for supporting reStructuredText as an input format</dd><dt>pytz</dt><dd>for timezone definitions</dd><dt>blinker</dt><dd>an object-to-object and broadcast signaling system</dd><dt>unidecode</dt><dd>for <span class="caps">ASCII</span> transliterations of Unicode text</dd></dl></div>
<p>It should print out a load of progress stuff and eventually finish by saying:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">Successfully installed pelican feedgenerator jinja2 pygments docutils pytz blinker unidecode six
Cleaning up...</code></pre></div>
<p>Double check it worked by running <code>pelican --version</code> - currently this should print out <code>3.2.0</code> - then run <code>pip freeze</code> - which prints out a list of the python modules installed in your current virtualenv.</p>
<p>Now we’ll install some extra python modules to support bonus functionality provided by some Pelican plugins that we’ll be using later:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pip <span class="nb">install </span>Pillow beautifulsoup4 cssmin cssprefixer cssutils pretty six smartypants typogrify webassets</code></pre></div>
<p>Again, as far as I know <span class="caps">PIL</span>/Pillow is hard to install on Windows - use the ActiveState Package Manager. Once this is done, run this to get <code>pip</code> to make a list of all the things you’ve got installed in this virtualenv:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pip freeze <span class="o">></span> requirements.txt</code></pre></div>
<p>Which should create a text file containing something like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">Jinja2==2.6
Pygments==1.6
Unidecode==0.04.12
argparse==1.2.1
blinker==1.2
docutils==0.10
feedgenerator==1.5
pelican==3.2
pytz==2013b
six==1.3.0
wsgiref==0.1.2</code></pre></div>
<p>This allows you to re-install everything in one go if you move machines, just by running <code>pip install -r requirements.txt</code> – or to check for <span class="amp">&</span> install updates to all the modules at once, just by running <code>pip install --upgrade -r requirements.txt</code>, amongst other things. We’re also going to check this lot into <code>git</code> later and this allows you to keep the list of requirements under version control too, which is nice.</p></section>
<section class="doc-section level-2"><h3 id="_pelican_quick_start">Pelican Quick Start</h3><p>Now that we’ve got everything installed, run this to create a basic skeleton site for you to modify:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>pelican-quickstart</code></pre></div>
<p>This will ask you some questions and generate a skeleton site, that matches your answers:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="go">Welcome to pelican-quickstart v3.2.0.
This script will help you create a new Pelican-based website.
Please answer the following questions so this script can generate the files needed by Pelican.
Using project associated with current virtual environment. Will save to:
/home/duncan/dev/duncanlock.net-pelican</span></code></pre></div>
<p>you can accept the defaults by pressing enter for most of these questions, except these:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">> What will be the title of this web site?
duncanlock.net
> Who will be the author of this web site?
Duncan Lock</code></pre></div>
<p>If you wanted to use the built-in Pelican webserver for development, you could say ‘No’ and skip this next bit, but we’re going to configure a local virtualhost and use Apache to serve the site for development, so we’re going to do this instead:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">> Do you want to specify a URL prefix? e.g., http://example.com (Y/n) y
> What is your URL prefix? (see above example; no trailing slash) http://duncanlock.test
[...]
Done. Your new project is available at /home/duncan/dev/duncanlock.net-pelican</code></pre></div>
<p>Now you can generate the quick-start site and see what it looks like:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>make html</code></pre></div>
<p>This should create an <code>output</code> folder with the contents of a website in it. To quickly serve the generated site so it can be previewed in your browser, run this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>make serve</code></pre></div>
<p>Then visit <a href="http://localhost:8000">http://localhost:8000</a> in your browser; you should be able to see a test site, which should look something like this:</p>
<div class="image-block"><img src="https://duncanlock.net/images/posts/how-i-built-this-website-using-pelican-part-1-setup/duncanlock-net-pelican-test.png" alt="Screenshot of the quick-started Pelican site, using the default theme and no content."></div>
<p>Press <code>Ctrl + c</code> in the console to stop the Pelican server.</p></section>
<section class="doc-section level-2"><h3 id="_apache_setup">Apache Setup</h3><p>Okay, now we want to configure an Apache VirtualHost <a class="footnote-ref" id="_footnoteref_8" href="#_footnote_8" title="View footnote 8" role="doc-noteref">[8]</a>, so that when we visit <a href="http://duncanlock.test/">http://duncanlock.test/</a> in a browser, our local Apache server will serve up our local pelican development site. There are lots of reasons why this is useful, but the main one is that it’s very close to my final deployment environment - a Linux box with Apache on it. It also means that the root of the local site is <code>/</code>, the same as the root of the final live site, which is nice for making links work. This also allows us to do neat server configuration things and test them all locally, as we’ll see later.</p>
<p>If you haven’t already got Apache installed, install it:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>apache2</code></pre></div>
<p>Once that’s finished, save the following as a text file called <code>duncanlock.test</code> in <code>/etc/apache2/sites-available/</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="apacheconf"># domain: duncanlock.test
<VirtualHost *:80>
# Admin email, Server Name (domain name) and any aliases
ServerAdmin webmaster@duncanlock.test
ServerName duncanlock.test
ServerAlias www.duncanlock.test
# Index file and Document Root (where the public files are located)
DirectoryIndex index.php index.html
DocumentRoot /home/duncan/dev/duncanlock.net-pelican/output/
</VirtualHost></code></pre></div>
<p>The really crucial bit of this is the <code>DocumentRoot</code> - make sure this points to the <code>/output/</code> folder of the Pelican site we just created - and use an absolute path.</p>
<p>Then add a mapping for the duncanlock.test domain to your <code>/etc/hosts</code> file, by adding this line somewhere:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">127.0.0.1 duncanlock.test</code></pre></div>
<p>Then enable our new virtual host in Apache:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>a2ensite duncanlock.test
<span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>service apache2 reload</code></pre></div>
<p>Now visiting <a href="http://duncanlock.test/">http://duncanlock.test/</a> in a browser should show your local Pelican development site.</p></section></section>
<section class="doc-section level-1"><h2 id="_git">Git</h2><p>It’s about time we started keeping some history of what we’re doing, so we will add our work so far to <code>git</code> <a class="footnote-ref" id="_footnoteref_9" href="#_footnote_9" title="View footnote 9" role="doc-noteref">[9]</a> - a version control system that will keep a history of all our changes, allow easy backups and restore, moving between machines, rolling back changes - and <em>much</em> more.</p>
<p>First, create a text file called <code>.gitignore</code> in your website’s root folder, containing this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="text">output/*
*.py[cod]</code></pre></div>
<p>This tells git to ignore everything in the output folder, and any compiled python files - we don’t need to version or backup that stuff.</p>
<p>Next, turn the current folder into a git repository and add our site so far:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>git init
<span class="go">
Initialized empty Git repository in /home/duncan/dev/duncanlock.net-pelican/.git/
</span><span class="gp">$</span><span class="w"> </span>git add <span class="nb">.</span>
<span class="gp">$</span><span class="w"> </span>git status
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>On branch master
<span class="gp">#</span><span class="w">
</span><span class="gp">#</span><span class="w"> </span>Initial commit
<span class="gp">#</span><span class="w">
</span><span class="gp">#</span><span class="w"> </span>Changes to be committed:
<span class="gp">#</span><span class="w"> </span><span class="o">(</span>use <span class="s2">"git rm --cached <file>..."</span> to unstage<span class="o">)</span>
<span class="gp">#</span><span class="w">
</span><span class="gp">#</span><span class="w"> </span>new file: .gitignore
<span class="gp">#</span><span class="w"> </span>new file: Makefile
<span class="gp">#</span><span class="w"> </span>new file: develop_server.sh
<span class="gp">#</span><span class="w"> </span>new file: pelicanconf.py
<span class="gp">#</span><span class="w"> </span>new file: publishconf.py
<span class="gp">#</span><span class="w"> </span>new file: requirements.txt
<span class="gp">#</span><span class="w">
</span><span class="go">
</span><span class="gp">$</span><span class="w"> </span>git commit <span class="nt">-m</span><span class="s2">"Inital commit of duncanlock.net; quick start site with no changes, so far"</span>
<span class="gp">$</span><span class="w"> </span>git status
<span class="go">
</span><span class="gp">#</span><span class="w"> </span>On branch master
<span class="go">nothing to commit, working directory clean</span></code></pre></div>
<p>That’s it - the site is now in git, ready to be backed up onto <a href="https://github.com/">GitHub</a>, if you like. When you make changes, remember to do the following, so they’re stored and versioned in git:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>git add <span class="nb">.</span>
<span class="gp">$</span><span class="w"> </span>git commit <span class="nt">-m</span><span class="s2">"Description of the changes I made."</span></code></pre></div>
<p><span class="caps">OK</span>, that’s it for part one - you should now have a working Pelican site, in a python virtual environment, being served by Apache via a VirtualHost on your local machine.</p></section>
<section class="doc-section level-1"><h2 id="_coming_up_in_part_2">Coming up in Part 2:</h2><div class="ulist"><ul><li>Content creation work-flow</li><li>Creating <span class="amp">&</span> customizing your theme</li><li>Custom Jinja filters</li><li>Configuring your Pelican site<ul><li>Date based post URLs: <code>/blog/2013/05/03/post-title-goes-here/</code></li><li>Plugins</li><li>Extra files to copy over</li><li>Twitter Cards</li><li>Favicons, sitemaps, Google Analytics,</li><li>etc…​</li></ul></li><li>Performance: Web assets - minifying <span class="amp">&</span> compressing things, professional Apache .htaccess setup</li><li>Deploying your site to your server</li></ul></div>
<p>Once I’ve finished part 2, I’ll link it here. If you’ve got any questions, please ask in the comments.</p>
<hr>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References:</h3></section></section><section class="footnotes" aria-label="Footnotes" role="doc-endnotes"><hr><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote">*Markdown* is a text-to-<span class="caps">HTML</span> conversion tool for web writers. Markdown allows you to write using an easy-to-read, easy-to-write plain text format, then convert it to structurally valid <span class="caps">XHTML</span> (or <span class="caps">HTML</span>): <a href="http://daringfireball.net/projects/markdown/">http://daringfireball.net/projects/markdown/</a> <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">*reStructuredText* is an easy-to-read, what-you-see-is-what-you-get plain text mark-up syntax and parser system. It is useful for in-line program documentation (such as Python docstrings), for quickly creating simple web pages, and for standalone documents: <a href="http://en.wikipedia.org/wiki/ReStructuredText">http://en.wikipedia.org/wiki/ReStructuredText</a> <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_3" role="doc-endnote">*AsciiDoc* is a text document format for writing notes, documentation, articles, books, ebooks, slideshows, web pages, man pages and blogs. AsciiDoc files can be translated to many formats including <span class="caps">HTML</span>, <span class="caps">PDF</span>, <span class="caps">EPUB</span>, man page: <a href="http://www.methods.co.nz/asciidoc/">http://www.methods.co.nz/asciidoc/</a> <a class="footnote-backref" href="#_footnoteref_3" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_4" role="doc-endnote">*SublimeText* is currently my <a href="http://www.sublimetext.com/">favourite text editor</a> - it’s really pretty great, you should try it. <a class="footnote-backref" href="#_footnoteref_4" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_5" role="doc-endnote">*Pip* is a package management system used to install and manage software packages written in the programming language Python. Many packages can be found in the Python Package Index (PyPI): <a href="http://en.wikipedia.org/wiki/Pip_(Python">http://en.wikipedia.org/wiki/Pip_(Python</a>) <a class="footnote-backref" href="#_footnoteref_5" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_6" role="doc-endnote">*virtualenv* is a tool to create isolated Python environments: <a href="http://www.virtualenv.org/en/latest/">http://www.virtualenv.org/en/latest/</a> <span class="amp">&</span> <a href="http://www.clemesha.org/blog/modern-python-hacker-tools-virtualenv-fabric-pip/">http://www.clemesha.org/blog/modern-python-hacker-tools-virtualenv-fabric-pip/</a> <a class="footnote-backref" href="#_footnoteref_6" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_7" role="doc-endnote">*virtualenvwrapper* is a set of extensions to Ian Bicking’s `virtualenv` tool. Includes wrappers for creating <span class="amp">&</span> deleting virtual environments and managing development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies. <a href="http://virtualenvwrapper.readthedocs.org/en/latest/">http://virtualenvwrapper.readthedocs.org/en/latest/</a> <a class="footnote-backref" href="#_footnoteref_7" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_8" role="doc-endnote">The Apache Webserver can serve lots of different websites from the same server instance, on the same <span class="caps">IP</span> address. Virtual Hosts are the way it does this. You just give each one a name, a folder and a mapping in your /etc/hosts files and reload Apache. <a class="footnote-backref" href="#_footnoteref_8" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_9" role="doc-endnote">*Git* is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency: <a href="http://git-scm.com/">http://git-scm.com/</a> <a class="footnote-backref" href="#_footnoteref_9" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>Using UDF as an improved filesystem for USB Flash Drives2013-05-13T19:48:22-07:002021-03-12T22:57:05-08:00Duncan Locktag:duncanlock.net,2013-05-13:/blog/2013/05/13/using-udf-as-an-improved-filesystem-for-usb-flash-drives/<p>Ever tried to copy something onto a <span class="caps">USB</span> flash drive, only to discover that the file was too big to copy?</p>
<p>This is because most <span class="caps">USB</span> Flash drives are formatted using the <span class="caps">FAT32</span><a class="footnote-ref" href="#_footnote_1" id="_footnoteref_1" role="doc-noteref" title="View footnote 1">[1]</a> filesystem - which only supports individual files up to 4 <span class="caps">GB</span> in size, no matter how much free space you’ve got. It also only supports drives up to 2 <span class="caps">TB</span>, can’t store symbolic links, can’t store files with these characters in the name: <code>"*/:<>?\|</code> – and is generally pretty crappy.</p>
<section class="doc-section level-1"><h2 id="_the_solution_is_to_use_a_better_filesystem">The solution is to use a better filesystem</h2><p>There are <a href="http://en.wikipedia.org/wiki/Comparison_of_file_systems">many filesystems</a> to choose from - but there are a few criteria that a good <span class="caps">USB</span> flash drive filesystem needs to meet which cut down the choices quite a bit:</p>
<div class="ulist"><ul><li>Compatible with every computer - just plug it in, and it should work</li><li>Has no permissions, or optional …</li></ul></div></section><p>Ever tried to copy something onto a <span class="caps">USB</span> flash drive, only to discover that the file was too big to copy?</p>
<p>This is because most <span class="caps">USB</span> Flash drives are formatted using the <span class="caps">FAT32</span><a class="footnote-ref" href="#_footnote_1" id="_footnoteref_1" role="doc-noteref" title="View footnote 1">[1]</a> filesystem - which only supports individual files up to 4 <span class="caps">GB</span> in size, no matter how much free space you’ve got. It also only supports drives up to 2 <span class="caps">TB</span>, can’t store symbolic links, can’t store files with these characters in the name: <code>"*/:<>?\|</code> – and is generally pretty crappy.</p>
<section class="doc-section level-1"><h2 id="_the_solution_is_to_use_a_better_filesystem">The solution is to use a better filesystem</h2><p>There are <a href="http://en.wikipedia.org/wiki/Comparison_of_file_systems">many filesystems</a> to choose from - but there are a few criteria that a good <span class="caps">USB</span> flash drive filesystem needs to meet which cut down the choices quite a bit:</p>
<div class="ulist"><ul><li>Compatible with every computer - just plug it in, and it should work</li><li>Has no permissions, or optional permissions, so you don’t need to be logged in as the same user everywhere you want to access the files</li><li>Can handle large files and volumes, special characters and is reliable - is a proper modern filesystem</li><li>Ideally, is an open standard, isn’t patent encumbered - or at the very least has a solid open implementation.</li></ul></div></section>
<section class="doc-section level-1"><h2 id="_a_better_filesystem_the_candidates">A better filesystem: The Candidates</h2><div class="table-block"><table class="frame-all grid-all" style="width: 84%;"><colgroup><col style="width: 24%;"/><col style="width: 21%;"/><col style="width: 22%;"/><col style="width: 14%;"/><col style="width: 19%;"/></colgroup><thead><tr><th class="halign-left valign-top">Filesystem</th><th class="halign-left valign-top">Compatible</th><th class="halign-left valign-top">Permissions</th><th class="halign-left valign-top">Modern(ish)</th><th class="halign-left valign-top">Open(ish)</th></tr></thead><tbody><tr><td class="halign-left valign-top">ext2,3,4</td><td class="halign-left valign-top">Linux</td><td class="halign-left valign-top">✗</td><td class="halign-left valign-top">✓</td><td class="halign-left valign-top">✓</td></tr><tr><td class="halign-left valign-top">hfs+</td><td class="halign-left valign-top">Linux, MacOS</td><td class="halign-left valign-top">✗</td><td class="halign-left valign-top">✓</td><td class="halign-left valign-top">(✓)</td></tr><tr><td class="halign-left valign-top">exFAT</td><td class="halign-left valign-top">✓<a class="footnote-ref" href="#_footnote_2" id="_footnoteref_2" role="doc-noteref" title="View footnote 2">[2]</a></td><td class="halign-left valign-top">✗</td><td class="halign-left valign-top">(✓)</td><td class="halign-left valign-top">(✓)</td></tr><tr><td class="halign-left valign-top">ntfs</td><td class="halign-left valign-top">✓<a class="footnote-ref" href="#_footnote_3" id="_footnoteref_3" role="doc-noteref" title="View footnote 3">[3]</a></td><td class="halign-left valign-top">✗</td><td class="halign-left valign-top">✓</td><td class="halign-left valign-top">(✓)</td></tr><tr><td class="halign-left valign-top">fat32</td><td class="halign-left valign-top">✓</td><td class="halign-left valign-top">✓</td><td class="halign-left valign-top">✗</td><td class="halign-left valign-top">(✓)</td></tr><tr><td class="halign-left valign-top">udf</td><td class="halign-left valign-top">✓</td><td class="halign-left valign-top">✓</td><td class="halign-left valign-top">✓</td><td class="halign-left valign-top">✓</td></tr></tbody></table></div>
<p>So, the winner is… <span class="caps">UDF</span><a class="footnote-ref" href="#_footnote_4" id="_footnoteref_4" role="doc-noteref" title="View footnote 4">[4]</a>. This filesystem is mostly used for <span class="caps">DVD</span>’s - but supports full random access read/write storage, like any general purpose filesystem. Pretty much all <span class="caps">OS</span>’s can read <span class="caps">DVD</span>’s out of the box, and it supports large files and volumes. Every <span class="caps">OS</span> since Windows 95 can read <span class="caps">UDF</span> natively, and almost all of them can write to it too.</p>
<p>The fly in the ointment, as usual, is older versions of Windows. Windows 95–2003 <strong>can’t write to <span class="caps">UDF</span> volumes natively</strong> - you need extra 3rd party software. Windows Vista, 7 <span class="amp">&</span> 8 <strong>can</strong> both read and write to <span class="caps">UDF</span> volumes, though. See <a href="http://en.wikipedia.org/wiki/Universal_Disk_Format#Compatibility">here for a full list of <span class="caps">OS</span> compatibility</a>. Microsoft are now calling <span class="caps">UDF</span> the ‘Live File System’<a class="footnote-ref" href="#_footnote_5" id="_footnoteref_5" role="doc-noteref" title="View footnote 5">[5]</a> and are still assuming it’s just for <span class="caps">DVD</span>’s.</p></section>
<section class="doc-section level-1"><h2 id="_formatting_your_usb_flash_drive_with_udf">Formatting your <span class="caps">USB</span> Flash Drive with <span class="caps">UDF</span></h2><aside class="admonition-block note" role="note"><h6 class="block-title"><span class="title-label">Note: </span>Caution</h6><p>If you need to be able to write to your <span class="caps">USB</span> flash drive on older version of Windows, pre-Vista - <strong>then don’t do this</strong>!</p>
<p>Also, this will wipe <span class="amp">&</span> reformat your <span class="caps">USB</span> flash drive, so backup anything you want to keep on another drive first.</p></aside>
<p>To format your <span class="caps">USB</span> Flash drive using <span class="caps">UDF</span>, you need to do the following:</p>
<p>You’ll probably need to install <code>udftools</code> first:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>udftools</code></pre></div>
<p>then wipe the existing partition table, to prevent the drive being detected as <span class="caps">FAT</span>. Assuming your <span class="caps">USB</span> drive is <code>/dev/sdi</code>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo dd </span><span class="k">if</span><span class="o">=</span>/dev/zero <span class="nv">of</span><span class="o">=</span>/dev/sdi <span class="nv">bs</span><span class="o">=</span>1M <span class="nv">count</span><span class="o">=</span>1</code></pre></div>
<p>Then format it as <span class="caps">UDF</span>:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>mkudffs <span class="nt">-b</span> 512 <span class="nt">--media-type</span><span class="o">=</span>hd <span class="nt">--utf8</span> <span class="nt">--lvid</span><span class="o">=</span>DriveLabel <span class="nt">--vid</span><span class="o">=</span>DriveLabel <span class="nt">--fsid</span><span class="o">=</span>DriveLabel /dev/sdi</code></pre></div>
<p>Change <code>DriveLabel</code> to whatever you want to drive to be called. Linux doesn’t seem to take any notice of this, but apparently windows displays <code>lvid</code> as the drive label.</p>
<p>If you need out of the box compatibility back to Windows 95, you can add this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="go">--udfrev=0x0102</span></code></pre></div>
<p>otherwise you’ll get Windows <span class="caps">XP</span> and up <a class="footnote-ref" href="#_footnote_6" id="_footnoteref_6" role="doc-noteref" title="View footnote 6">[6]</a>.</p>
<p>The <code>-b 512</code> parameter forces a file system block size equal to the <span class="caps">USB</span> stick’s physical block size - as required by the <span class="caps">UDF</span> specification. It’ll <em>probably</em> work without this, but it’s a good idea to set this to ensure maximum compatibility <span class="amp">&</span> reliability. Change the <code>512</code> if you’re lucky enough to have a <span class="caps">USB</span> flash drive with a larger block size. To find out what the physical block size is, run this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>hdparm <span class="nt">-I</span> /dev/sdi | <span class="nb">grep</span> <span class="nt">-i</span> physical
<span class="c">...
</span><span class="go">Logical/Physical Sector size: 512 bytes</span></code></pre></div>
<p>This worked well for me on Xubuntu Linux: the performance is good, I was able to copy some virtual machines from one computer to another - a total of 11.5 <span class="caps">GB</span> of files, with individual files up to 6.5 <span class="caps">GB</span> each - without any problems.</p>
<p><a href="http://tanguy.ortolo.eu/blog/article93/usb-udf#c1359985488-1">There are some Mac instructions here</a> which I haven’t tried — and I have <em>no idea</em> how you do this on Windows; let me know in the comments if you do.</p>
<hr/>
<section class="doc-section level-2"><h3 id="_footnotes_references">Footnotes <span class="amp">&</span> References:</h3><p>I owe most of this to <a href="http://tanguy.ortolo.eu/blog/article93/usb-udf">Tanguy Ortolo’s excellent blog post which you can find here</a> <span class="amp">&</span> this <a href="https://bbs.archlinux.org/viewtopic.php?pid=1030147">Arch linux forum post</a>.</p></section></section><section aria-label="Footnotes" class="footnotes" role="doc-endnotes"><hr/><ol class="footnotes"><li class="footnote" id="_footnote_1" role="doc-endnote"><strong><span class="caps">FAT32</span></strong> - File Allocation Table (<span class="caps">FAT</span>) is the name of a computer file system architecture and a family of proprietary file systems utilizing it: <a class="bare" href="http://en.wikipedia.org/wiki/FAT32#FAT32">http://en.wikipedia.org/wiki/<span class="caps">FAT32</span>#<span class="caps">FAT32</span></a> <a class="footnote-backref" href="#_footnoteref_1" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_2" role="doc-endnote">For Linux, this isn’t included in the kernel, as it’s proprietary. You need to install <code>exfat-fuse</code> <span class="amp">&</span> <code>exfat-utils</code> for this to work. <a class="footnote-backref" href="#_footnoteref_2" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_3" role="doc-endnote">For Linux, this isn’t included in the kernel, as it’s proprietary. You need to install <code>ntfs-3g</code> for this to work <a class="footnote-backref" href="#_footnoteref_3" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_4" role="doc-endnote"><strong>Universal Disk Format (<span class="caps">UDF</span>)</strong> - is a profile of the specification known as <span class="caps">ISO</span>/<span class="caps">IEC</span> 13346 and <span class="caps">ECMA</span>-167 and is an open vendor-neutral file system for computer data storage for a broad range of media: <a class="bare" href="http://en.wikipedia.org/wiki/Universal_Disk_Format">http://en.wikipedia.org/wiki/Universal_Disk_Format</a> <a class="footnote-backref" href="#_footnoteref_4" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_5" role="doc-endnote"><strong>Which <span class="caps">CD</span> or <span class="caps">DVD</span> format should I use?</strong>: <a class="bare" href="http://windows.microsoft.com/en-us/windows-vista/which-cd-or-dvd-format-should-i-use">http://windows.microsoft.com/en-us/windows-vista/which-cd-or-dvd-format-should-i-use</a> <a class="footnote-backref" href="#_footnoteref_5" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li><li class="footnote" id="_footnote_6" role="doc-endnote"><strong><span class="caps">UDF</span> has Multiple revisions</strong>: <a class="bare" href="http://en.wikipedia.org/wiki/Universal_Disk_Format#Revisions">http://en.wikipedia.org/wiki/Universal_Disk_Format#Revisions</a> <a class="footnote-backref" href="#_footnoteref_6" role="doc-backlink" title="Jump to the first occurrence in the text">↩</a></li></ol></section>How to convert Apple Lossless/ALAC/.m4a files to FLAC with avconv, on Ubuntu Linux2013-05-07T20:12:59-07:002021-06-11T11:06:19-07:00Duncan Locktag:duncanlock.net,2013-05-07:/blog/2013/05/07/how-to-convert-apple-lossless-alac-m4a-files-to-flac-with-avconv-on-ubuntu-linux/<p>I recently needed to convert some Apple Lossless music files to <span class="caps">FLAC</span>. Here’s how to do it:</p>
<p>If you don’t already have <code>ffmpeg</code> or <code>libav-tools</code> installed, do this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>apt-get <span class="nb">install </span>libav-tools</code></pre></div>
<p>Then run this to do the conversion, in the folder with music in:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="k">for </span>f <span class="k">in</span> <span class="k">*</span>.m4a<span class="p">;</span> <span class="k">do </span>avconv <span class="nt">-i</span> <span class="s2">"</span><span class="nv">$f</span><span class="s2">"</span> <span class="s2">"</span><span class="k">${</span><span class="nv">f</span><span class="p">%.m4a</span><span class="k">}</span><span class="s2">.flac"</span><span class="p">;</span> <span class="k">done</span></code></pre></div>
<p>And that’s it - it will convert all the <code>.m4a</code> files in that folder to <code>.flac</code> files, preserving the metadata. You can now delete the <code>.m4a</code> files if you want:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span><span class="nb">rm</span> ./<span class="k">*</span>.m4a</code></pre></div>How to set your Compose Key on XFCE/Xubuntu & LXDE Linux2013-05-03T15:40:46-07:002021-06-11T19:40:21-07:00Duncan Locktag:duncanlock.net,2013-05-03:/blog/2013/05/03/how-to-set-your-compose-key-on-xfce-xubuntu-lxde-linux/<figure class="image-block"><img src="https://duncanlock.net/images/posts/how-to-set-your-compose-key-on-xfce-xubuntu-linux/compose-key-diagram.png" alt="Blueprint style diagram showing the compose key sequence for the Euro currency symbol.">
<figcaption>Figure 1. Just hold down your chosen compose key, then press the other keys in turn: [compose key] + e + = gets you a Euro symbol.</figcaption></figure>
<p>The compose key on Linux is <em>incredibly</em> useful, but not configured by default - and on <span class="caps">XFCE</span> there’s currently no graphical <span class="caps">UI</span> to change it. However, it’s pretty simple to change…​ here’s how to make the Caps Lock key your compose key:</p>
<p>In the file <code>/etc/default/keyboard</code>, change <code>XKBOPTIONS</code> to look like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="ini"><span class="py">XKBOPTIONS</span><span class="p">=</span><span class="s">"compose:caps"</span></code></pre></div>
<p>Save the file. You can now either reboot, restart X, or see <a href="#_activating_your_change_without_rebooting">Activating your change without rebooting</a>, below.</p>
<p>If you don’t want to use Caps Lock as your compose key, there are some other options to choose from in this file: <code>/usr/share/X11/xkb/rules/xorg.lst</code> - search for ‘compose’, they’re listed towards the …</p><figure class="image-block"><img src="https://duncanlock.net/images/posts/how-to-set-your-compose-key-on-xfce-xubuntu-linux/compose-key-diagram.png" alt="Blueprint style diagram showing the compose key sequence for the Euro currency symbol.">
<figcaption>Figure 1. Just hold down your chosen compose key, then press the other keys in turn: [compose key] + e + = gets you a Euro symbol.</figcaption></figure>
<p>The compose key on Linux is <em>incredibly</em> useful, but not configured by default - and on <span class="caps">XFCE</span> there’s currently no graphical <span class="caps">UI</span> to change it. However, it’s pretty simple to change…​ here’s how to make the Caps Lock key your compose key:</p>
<p>In the file <code>/etc/default/keyboard</code>, change <code>XKBOPTIONS</code> to look like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="ini"><span class="py">XKBOPTIONS</span><span class="p">=</span><span class="s">"compose:caps"</span></code></pre></div>
<p>Save the file. You can now either reboot, restart X, or see <a href="#_activating_your_change_without_rebooting">Activating your change without rebooting</a>, below.</p>
<p>If you don’t want to use Caps Lock as your compose key, there are some other options to choose from in this file: <code>/usr/share/X11/xkb/rules/xorg.lst</code> - search for ‘compose’, they’re listed towards the end. Mine currently looks like this:</p>
<div class="table-block"><table class="frame-all grid-all stretch"><colgroup><col style="width: 41%;"><col style="width: 59%;"></colgroup><thead><tr><th class="halign-left valign-top">Setting</th><th class="halign-left valign-top">Compose Key</th></tr></thead><tbody><tr><td class="halign-left valign-top">compose:ralt</td><td class="halign-left valign-top">Right Alt</td></tr><tr><td class="halign-left valign-top">compose:lwin</td><td class="halign-left valign-top">Left Win</td></tr><tr><td class="halign-left valign-top">compose:rwin</td><td class="halign-left valign-top">Right Win</td></tr><tr><td class="halign-left valign-top">compose:menu</td><td class="halign-left valign-top">Menu</td></tr><tr><td class="halign-left valign-top">compose:lctrl</td><td class="halign-left valign-top">Left Ctrl</td></tr><tr><td class="halign-left valign-top">compose:rctrl</td><td class="halign-left valign-top">Right Ctrl</td></tr><tr><td class="halign-left valign-top">compose:caps</td><td class="halign-left valign-top">Caps Lock</td></tr><tr><td class="halign-left valign-top">compose:102</td><td class="halign-left valign-top"><Less/Greater></td></tr><tr><td class="halign-left valign-top">compose:paus</td><td class="halign-left valign-top">Pause</td></tr><tr><td class="halign-left valign-top">compose:prsc</td><td class="halign-left valign-top">PrtSc</td></tr><tr><td class="halign-left valign-top">compose:sclk</td><td class="halign-left valign-top">Scroll Lock</td></tr></tbody></table></div>
<section class="doc-section level-1"><h2 id="_using_the_compose_key">Using the Compose Key</h2><p>Try holding down the Caps Lock key and type the letter <code>l</code>, followed by a dash you should get a British Pound symbol: £ - magic!</p>
<p>Generally you can just hold down your compose key then type a combo that kinda ‘looks like’ the symbol you want, if they were drawn on top of each other. For example, <code>e</code> followed by <code>=</code> will get you a Euro symbol: €, <code>o</code> followed by <code>c</code> will get you a copyright symbol: © - and <code>1</code> followed by <code>2</code> gets you a half fraction: ½.</p>
<p>These work anywhere you can type things. There’s a list of some common ones <a href="http://en.wikipedia.org/wiki/Compose_key#Common_compose_combinations">here</a>, and a <a href="http://www.hermit.org/Linux/ComposeKeys.html">full list here</a>.</p></section>
<section class="doc-section level-1"><h2 id="_activating_your_change_without_rebooting">Activating your change without rebooting</h2><section class="doc-section level-2"><h3 id="_option_one">Option One</h3><p>Once you’ve changed your permanent setup by changing <code>/etc/default/keyboard</code>, you can set your current session using <code>setxkbmap</code>, like this:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>setxkbmap <span class="nt">-option</span> compose:caps</code></pre></div>
<p>It uses the same names for keys as above. It’s also possible to leave <code>/etc/default/keyboard</code> alone and add this command to your <code>~/.xinit</code>, <code>~/.xsession</code> or the <span class="caps">XFCE</span> autostart commands instead. If you do that, it’ll only apply to your login, not system wide though.</p></section>
<section class="doc-section level-2"><h3 id="_option_two">Option Two</h3><p>This was the first method that I discovered and isn’t a simple as Option One, but if that doesn’t work for some reason, this should. Like it says at the top of <code>/etc/default/keyboard</code>, check <code>/usr/share/doc/keyboard-configuration/README.Debian</code> for what to do after you’ve modified it. Mine currently says:</p>
<div class="quote-block"><blockquote><p>After modifying <code>/etc/default/keyboard</code>, you can apply the changes to the Linux
console by running <code>setupcon</code>. If X is configured to use that file too, then the
changes will become visible to X only if</p>
<p><code>udevadm trigger --subsystem-match=input --action=change</code></p>
<p>is called, or the system is rebooted.</p></blockquote></div>
<p>When it says Console, it means it – not a terminal window, an actual login console – try pressing <code>Ctrl + Alt + F1</code> and logging in, then running:</p>
<div class="listing-block"><pre class="rouge highlight"><code data-lang="console"><span class="gp">$</span><span class="w"> </span>setupcon
<span class="gp">$</span><span class="w"> </span>udevadm trigger <span class="nt">--subsystem-match</span><span class="o">=</span>input <span class="nt">--action</span><span class="o">=</span>change</code></pre></div>
<p>These don’t print anything out when they run, disconcertingly. Once you’ve run them in the Console, <code>Ctrl + Alt + F7</code> should get you back to the default console session you were in before.</p></section></section>Welcome to the New Site; same as the Old Site.2013-04-26T16:48:57-07:002013-04-26T16:48:57-07:00Duncan Locktag:duncanlock.net,2013-04-26:/blog/2013/04/26/welcome-to-the-new-site-same-as-the-old-site/<p>I’ve been meaning to consolidate my personal websites onto this domain for a <em>long, long</em> time. My original personal website, <a href="http://www.dflock.co.uk/">dflock.co.uk</a>, started in the late nineties - and has been getting a bit long in the tooth of late.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/welcome-to-the-new-site-same-as-the-old-site/screenshot-13-04-26_06-54-42-pm.png" alt="Screenshot of the current version of the dflock.co.uk website homepage" width="at the time of publishing this post. It's kinda green and nineties looking.">
<figcaption>Figure 1. Static text files - the gift that keeps on giving.</figcaption></figure>
<p>That site has been up, looking mostly like that, for ~15 years - with zero maintenance or downtime. Nice to know it’s still valid <span class="caps">CSS</span> 1.0 :)</p>
<p>When I created that site originally, the options were either <a href="http://en.wikipedia.org/wiki/Adobe_PageMill">Adobe PageMill</a>, or hand editing <span class="caps">HTML</span> in a text editor. I’ve never been a fan of <code>WYSIWYG (what-you-see-is(supposedly)-what-you-get)</code> editors - and the early ones were…​ not good, so I wrote all the <span class="caps">HTML</span> code by hand.</p>
<p>That site also has some <a href="http://www.dflock.co.uk/colitis/foods/enumbers.html">data heavy pages</a>, which are manually published from …</p><p>I’ve been meaning to consolidate my personal websites onto this domain for a <em>long, long</em> time. My original personal website, <a href="http://www.dflock.co.uk/">dflock.co.uk</a>, started in the late nineties - and has been getting a bit long in the tooth of late.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/welcome-to-the-new-site-same-as-the-old-site/screenshot-13-04-26_06-54-42-pm.png" alt="Screenshot of the current version of the dflock.co.uk website homepage" width="at the time of publishing this post. It's kinda green and nineties looking.">
<figcaption>Figure 1. Static text files - the gift that keeps on giving.</figcaption></figure>
<p>That site has been up, looking mostly like that, for ~15 years - with zero maintenance or downtime. Nice to know it’s still valid <span class="caps">CSS</span> 1.0 :)</p>
<p>When I created that site originally, the options were either <a href="http://en.wikipedia.org/wiki/Adobe_PageMill">Adobe PageMill</a>, or hand editing <span class="caps">HTML</span> in a text editor. I’ve never been a fan of <code>WYSIWYG (what-you-see-is(supposedly)-what-you-get)</code> editors - and the early ones were…​ not good, so I wrote all the <span class="caps">HTML</span> code by hand.</p>
<p>That site also has some <a href="http://www.dflock.co.uk/colitis/foods/enumbers.html">data heavy pages</a>, which are manually published from <span class="caps">XML</span> files using <code>XSLT (Extensible Stylesheet Language Transformations)</code> to output <span class="caps">HTML</span>, processed by <a href="http://en.wikipedia.org/wiki/Saxon_XSLT">Saxon</a>, later <a href="http://en.wikipedia.org/wiki/XMLStarlet">XMLStarlet</a> – again, all written and run by hand.</p>
<p>As far as I can remember, I did all that on a 486sx Windows ‘98 <span class="caps">PC</span>, using the <a href="http://en.wikipedia.org/wiki/TextPad">TextPad</a> text editor – which is apparently still available, unlike Adobe PageMill.</p>
<section class="doc-section level-1"><h2 id="_baked_not_fried">Baked, not Fried</h2><p>Since then, I’ve been in the trenches doing web development, amongst other things. Along came <span class="caps">CSS</span>, then <span class="caps">PHP</span>, Dynamic Sites, MySQL, JavaScript (anyone remember <code>DHTML (Dynamic HTML)</code>?), <span class="caps">AJAX</span>, Web Apps, Websockets, Ruby, Python, Cloud Hosting, <span class="caps">HTML5</span>, etc, etc…​ things have got a <em>lot</em> more complex in the web development world over the last 15 years.</p>
<p>There are good reasons for most of these, obviously, but it unavoidably layers on a lot of complexity. Websites that were originally just a folder full of static files are now a complex cocktail of many different pieces of technology. A ‘simple blog’ nowadays might well combine <span class="caps">PHP</span>, MySQL, <span class="caps">HTML</span>, <span class="caps">CSS</span> <span class="amp">&</span> JavaScript – like <a href="http://wordpress.com/">WordPress</a> does, for example. All of these moving parts have to work together to deliver the site – and they all have to be bulletproof to keep it secure.</p>
<p>But in the end, browsers still mostly just display <span class="caps">HTML</span>, <span class="caps">CSS</span> <span class="amp">&</span> JavaScript - which they load from the server. How you generate that in the first place is up to you. You can bake it in advance, or fry it up on each page request. The most recent spate of widely publicized WordPress security exploits might have tipped me over the edge - I decided that I wanted to simplify.</p>
<p>If I was going to update my old sites, I was going to do it Old School - static files, lean <span class="amp">&</span> clean.</p></section>
<section class="doc-section level-1"><h2 id="_so_here_we_are_full_circle">So, here we are, full circle</h2><p>I’m sitting writing this post in <a href="http://docutils.sourceforge.net/rst.html">ReStructuredText</a> - a simple, easy-to-read, what-you-see-is-what-you-get plaintext format. I’m using an amazing text editor - <a href="http://www.sublimetext.com/">SublimeText</a> - on a quad-core, <span class="caps">8GB</span> <span class="caps">RAM</span>, Linux box. Every time I hit <span class="caps">CTRL</span>+s, <a href="http://docs.getpelican.com/">Pelican</a> automatically pours my words into the site’s templates (written in <a href="http://jinja.pocoo.org/">Jinja</a>) and bakes it all into static files in a folder. It’ll then upload it to my server if I want, or I can upload them myself.</p>
<figure class="image-block"><img src="https://duncanlock.net/images/posts/welcome-to-the-new-site-same-as-the-old-site/screenshot-13-04-28_12-48-16-pm.png" alt="Screenshot of the above paragraph being written in SublimeText.">
<figcaption>Figure 2. Just write, save and a website appears, as if by magic.</figcaption></figure>
<p>Static text files - they really are the gift that keeps on giving.</p>
<p>The same, but different. The same, but better.</p></section>