• Homegrown Backups

    I mentioned a while back that I’d moved to using my NUC as a backup storage device, and that continues to be a core use case after I repaved and moved the thing back over to Ubuntu.

    Fortunately, as a file server, Linux is definitely more capable and compatible than macOS (which is why, back when it was a Hackintosh, I used a Linux VM as the SMB implementation on my LAN), and so I’ve already got backups re-enabled and working beautifully.

    But the next step is enabling offsite copies.

    Previously, I achieved this with Google Drive for macOS, backing up the backup directory to the cloud, a solution which worked pretty well overall! Unfortunately, Google provides no client for Linux, which left me in a bit of a jam.

    Until I discovered the magic that is rclone.

    rclone is, plain and simply, a command-line interface to cloud storage platforms. And it’s an incredibly capable one! It supports one-way folder synchronization (it doesn’t support two-way, but fortunately I don’t need that capability), which means that it’s the perfect solution for syncing up a local backup folder to an offsite cloud stored backup.

    But wait, there’s more!

    rclone also supports encryption. And that means that (assuming I don’t lose the keys… they’re safely stored in my keepass database (which, itself, is cloned in multiple locations using my other favourite tool, Syncthing)) I can protect those offsite backups from prying eyes, something which Google’s Drive sync tool does not offer.

    I can also decide when I want the synchronization to occur! I don’t need offsites done daily. Weekly would be sufficient, and that’s a simple crontab entry away.

    Now, to be clear, rclone would have worked just as well on the Hackintosh, so if you’re a Mac user who’d like to take advantage of rclone’s capabilities, you can absolutely do so! But for this Linux user, it was a pleasant surprise!

  • Fun with Puppeteer

    In the past web scraping involved a lot of offline scripting and parsing of HTML, either through a library or, for quick and dirty work, manual string transformations. The work was always painful, and as the web has become more dynamic, this offline approach has gone from painful to essentially impossible… you simply cannot scrape the contents of a website without a Javascript engine and a DOM implementation.

    The next generation of web scraping came in the form of tools like Selenium. Selenium uses a scripting language, along with a browser-side driver, to automate browser interactions. The primary use case for this particular stack is actually web testing, but it allows scraping by taking advantage of a full browser to load dynamic content. This allows you to simulate human interactions with the site, enabling scraping of even the most dynamic sites out there.

    Then came PhantomJS. PhantomJS took browser automation to the next level by wrapping a headless browser engine in a Javascript API. Using Javascript, you could then instantiate a browser, load a site, and interact with the page using standard DOM APIs. No longer did you need a secondary scripting language or a browser driver… in fact, you didn’t even need a GUI! Again, one of the primary use cases for this kind of technology is testing, but site automation in general, and scraping in particular, are excellent use cases for Phantom.

    And then the Chrome guys came along and gave us Puppeteer.

    Puppeteer is essentially PhantomJS but using the Chromium browser engine, delivered as an npm you can run atop node. Current benchmarks indicate Puppeteer is faster and uses less memory while using a more up-to-date browser engine.

    You might wonder why I started playing with Puppeteer.

    Well, it turns out Google Groups is sitting on a pretty extensive archive of old Usenet posts, some of which I’ve written, all of which date back to as early as ‘94. I wanted to archive those posts for myself, but discovered Groups provides no mechanism or API for pulling bulk content from their archive.

    For shame!

    Fortunately, Puppeteer made this a pretty easy nut to crack, such that it was just challenging enough to be fun, but easy enough to be done in a day. And thus I had the perfect one-day project during my holiday! The resulting script is roughly 100 lines of Javascript that is mostly reliable (unless Groups takes an unusually long time loading some of its content):

    const puppeteer = require('puppeteer')
    const fs = require('fs')
    async function run() {
      var browser = await puppeteer.launch({ headless: true });
      async function processPage(url) {
        const page = await browser.newPage();
        await page.goto(url);
        await page.addScriptTag({url: 'https://code.jquery.com/jquery-3.2.1.min.js'});
        await page.waitForFunction('$(".F0XO1GC-nb-Y").find("[dir=\'ltr\']").length > 0');
        await page.waitForFunction('$(".F0XO1GC-nb-Y").find("._username").text().length > 0');
        await page.exposeFunction('escape', async () => {
        await page.exposeFunction('log', async (message) => {
        var messages = await page.evaluate(async () => {
          function sleep(ms) {
            return new Promise(resolve => setTimeout(resolve, ms));
          var res = []
          await sleep(5000);
          var messages = $(".F0XO1GC-nb-Y");
          var texts = messages.find("[dir='ltr']").filter("div");
          for (let msg of messages.get()) {
            // Open the message menu
            await sleep(100);
            // Find the link button
            await sleep(100);
            // Grab the URL
            var msgurl = $(".F0XO1GC-Cc-b").filter("input").val().replace(
            ).replace("msg/", "msg=");
            await sleep(100);
            // Now close the thing
            var text;
            await $.get(msgurl, (data) => text = data);
              'username': $(msg).find("._username").text(),
              'date': $(msg).find(".F0XO1GC-nb-Q").text(),
              'url': msgurl,
              'message': text
            window.log("Message: " + res.length);
          return JSON.stringify({
            'group': $(".F0XO1GC-mb-x").find("a").first().text(),
            'count': res.length,
            'subject': $(".F0XO1GC-mb-Y").text(),
            'messages': res
          }, null, 4);
        await page.close();
        return messages;
      for (let url of urls) {
        var parts = url.split("/");
        var id = parts[parts.length - 1];
        console.log("Loading URL: " + url);
        fs.writeFile("messages/" + id + ".json", await processPage(url), function(err) {
          if (err) {
            return console.log(err);

    The interactions, here, are actually fairly complex. Each Google Groups message has a drop-down menu that you can use to get a link to the message itself. Some minor transformations to that URL then get you a link to the raw message contents. So this script loads the URL containing the thread, and then one-by-one, opens the menu, activates the popup to get the link, performs an Ajax call to get the message content, then scrapes out some relevant metadata and adds the result to a collection. The collection is then serialized out to JSON.

    It works remarkably well for a complete hack job!