When I heard about this tax holiday plan, my brain went to the tactical aspects of it. How do millions of Canadian businesses reprogram their POS to remove tax from an arbitrary list of items? And is that list even specific enough to avoid arguments on the til?

Then I became aware that for HST provinces, the holiday would remove the provincial portion of the tax. I don’t know how those provinces are supposed to make up literally millions, if not billions, of tax dollars. And how can it be legal for the federal government to just cut that money off with no discussion or recourse?

I’ve voted Liberal or NDP my entire adult life, but even I am having trouble with the ill-conceived stuff coming out of Trudeau’s office these days.

#politics

my shorter content on the fediverse: https://cosocial.ca/@jonw

Unlike many people, I don’t use a VPN to access geo-restricted content – AKA, “the Netflix reason”. I use a VPN for a two different reasons:

1. To ensure my internet traffic is encrypted even when I am using services or sites that do not provide encryption natively,.

2. To deny my ISP and their “partners” access to my internet traffic.

In short, I use a VPN to provide privacy, but I do not care about anonymity. If you’re not steeped in privacy terminology, the difference between anonymity and privacy is this:

Anonymity means I can see what you’re doing, but I don’t know who you are. Privacy means I can’t see what you’re doing, but I know who you are. All VPNs provide some level of privacy because they all encrypt traffic between your device and the VPN server. Most VPNs also provide anonymity incidentally because many VPN customers use the same VPN server and therefore traffic between those users is intermingled, and it’s not possible to determine which customer belongs to which traffic.

Note: many free or unethical VPN providers observe your traffic on their server but that is a business decision and not a shortcoming in the VPN architecture itself. When considering a VPN provider, read their privacy policy and terms of service. Look for words like “we do not capture logs” and stuff like that.

The recurring problems I have with VPN providers are:

• VPN does not stay connected

• VPN is not supported on one or more of my devices (iOS, Android, Linux)

• VPN will not resume after a suspend (laptop problem, specifically)

• VPN is too damn expensive

• Support is terrible

I’ve used a lot of different VPN services over the years while trying to find one that doesn’t have any of those problems and I’ve never been able to find one. They all have one or more of those problems.

Once I realized that I was tilting at windmills, I turned my attention inward to the things I know. I know how to use Secure SHell (SSH) – as a Linux sysadmin it is my main tool to connect to remote systems. I also know how to make SSH run as a proxy so I can tunnel internet traffic through it. I also have many servers out on the internet which I can use. However, the problem with using SSH as a proxy is that you have to configure your operating system to use the proxy. Or, at least, individual applications need to support proxy configuration. Spoiler: most do not.

Configuring an OS to use a proxy is easy. All the major OSes can be configured to use a proxy somewhere in their network settings. But dislike that option because there are some sites I use that will block connections from commercial IP addresses, or have some other objection when I try to log into my account from a distant IP address. Because of that, I need to shut off the proxy temporarily sometimes and digging through my OS innards to do that is a pain.

Setting up individual apps to use an SSH is a hit-and-miss because not all applications have proxy settings. FoxyProxy is a plugin for both Chrome and Firefox that makes SSH proxying through a browser trivial, but there’s just a truckload of internet traffic on our computers that has nothing to do with any visible app we’re running and it’s not possible to track it all down to its respective app and then see if that app can be configured to use the SSH proxy. That’s a losing battle and running a VPN that tunnels all network traffic, regardless of origin, through the VPN is the best solution.

During my journey to find the perfect VPN I never took my eyes off SSH. I wanted to find a solution that uses SSH because it is reliable, mature, and widely available but I had discarded it as impractical for the reasons I mentioned in the previous paragraph. Then, out of the blue, I ran across sshuttle.

What is sshuttle?

I was very pleased to find the sshuttle project which does exactly what want. It uses SSH as a full-blown VPN and routes all my traffic through the SSH tunnel just as a VPN would, but without all the crappy things about an actual VPN. The sshuttle maintainers totally get me as shown by this list of reasons why sshuttle is better than a VPN.

As far as I know, sshuttle is the only program that solves the following common case:

• Your client machine (or router) is Linux, FreeBSD, or MacOS.

• You have access to a remote network via ssh.

• You don't necessarily have admin access on the remote network.

• The remote network has no VPN, or only stupid/complex VPN protocols (IPsec, PPTP, etc). Or maybe you are the admin and you just got frustrated with the awful state of VPN tools.

• You don't want to create an ssh port forward for every single host/port on the remote network.

• You hate openssh's port forwarding because it's randomly slow and/or stupid.

• You can't use openssh's PermitTunnel feature because it's disabled by default on openssh servers; plus it does TCP-over-TCP, which has terrible performance.

How do I use sshuttle?

First, install it! It is in most repositories I’ve checked. You can install it on Debian strains like so:

sudo apt install sshuttle

The only other thing you need is access to a shell account on some remote server, usually a Linux server. Assuming you have that, you can use this quick one-liner to get the tunnel running:

sudo sshuttle -r $USER@$SERVER -x $SERVER 0/0

That’s it. Once you log in, the tunnel is set up and you’re now tunneling all your traffic through the remote server. Try this to see how your IP has changed:

curl ipinfo.io

How do I use sshuttle better?

I wanted a little more than the basics. I do not like to use SSH password authentication so I needed sshuttle to take a keypair to log in. Thankfully, you can send sshuttle standard ssh options by using the sshcmd switch. I use the sshcmd switch to tell sshuttle to use a key to authenticate to my remote server like this:

sudo sshuttle -r $USER@$SERVER:$PORT -x $SERVER 0/0 —ssh-cmd 'ssh -i /$PATH/$SSH_KEY'

Pro tip: If you automate the connection via systemd (more on that later) then you do not want a passphrase on your private key. I most definitely do have a passphrase on my privileged user key and I do not want to remove it. To get around this, I created a non-priveleged user that cannot sudo on my system and created a passwordphrase-less keypair to it. I use that account/keypair for my sshuttle use.

Next, I wanted to send my DNS requests through the SSH tunnel in addition to my traffic. Many people forget this step when setting up privacy tools. You may know that DNS is the Domain Name System, and it is responsible for converting domain names like jonwatson.ca to IP addresses like 192.124.249.64. Armed with that knowledge, you can see how allowing your ISP to resolve your DNS requests provides your ISP with a nice list of every site you’ve ever visited. Pushing those DNS requests through the tunnel to your remote server eliminates your ISP’s ability to observe those requests. To add this protection to my sshuttle, I add the --dns switch:

sudo sshuttle -r $USER@$SERVER:$PORT -x $SERVER 0/0 —dns —ssh-cmd 'ssh -i /$PATH/$SSH_KEY'

You can confirm your DNS is being pushed through your remote server by using a DNS Leak test site like this one. For example, my remote server is configured to use OpenDNS in the /etc/resolv.conf file, and that is what I see when I run a DNS Leak test, not my ISP’s or local network resolvers.

The final thing I wanted out of sshuttle was to resume the tunnel after my laptop comes out from suspend. It is my experience that this does not work well with any VPN providers. However, in those cases, I could usually track it down to a bug in OpenVPN. Most VPN providers are running OpenVPN under the hood, and when my laptop comes back from suspend, the OpenVPN manager is not responsive and therefore the VPN cannot restart without manual intervention. No such problem with sshuttle because it doesn’t use OpenVPN.

There are a few other options that may interest you, such as the --daemon option which tells sshuttle to fall to the background once it starts. I prefer to see all my juicy debug in the terminal as it runs so I do not use that option, but you may like it. Try the sshuttle —help command to see more options.

If you’re using a Linux distribution that uses systemd, this should work for you (replace the variables with your server and user, obviously):

$ cat /usr/lib/systemd/system-sleep/sshuttle #!/bin/sh case $1 in post) /usr/bin/sshuttle -r $USER@$SERVER:$PORT -x $SERVER 0/0 —dns —ssh-cmd 'ssh -i /$PATH/$SSH_KEY' ;; esac

If you’re using a SysV system, use your startup files in /etc/init.d to control the sshuttle startup

That solves my laptop and desktop VPN problem. It still does not solve my Android or iOS problem, but I can continue using a conventional VPN until I find a similarly elegant solution for those devices.

Questions or comments?

As a paid subscriber, you can comment and “like” my posts. Just click the “Like and Comment” button below.

my shorter content on the fediverse: https://cosocial.ca/@jonw

image courtesy of Slack

My company recently moved from Slack to Microsoft Teams as our core communications tool. It was inevitable, really. Slack pissed Microsoft off 4 years ago with its silly full-page ad in the New York Times and that decision has come back to haunt Slack. When you poke the sleeping dragon, the dragon responds by giving its competing product away to its BILLIONS of users for free. Goodbye Slack, that was dumb.

Pushing the business case aside, as a technical Slack user working remotely for a tech company, I’ve really felt the burn. Microsoft Teams is orders of magnitude less capable than Slack. Some of that inferiority is simply because Teams is feature-poor compared to Slack, but some of it is just plain old Microsoft incompetence in its inability to fix bugs that have existed in its chat clients for years.

Here are some of the things that make Teams a total stink bomb.

App integration…isn’t.

Microsoft either doesn’t understand what the word “integration” means, or doesn’t care. Let’s take a look at a useful PagerDuty integration we use at work.

We told our PagerDuty account to alert into a Slack room by toggling a few switches in our PagerDuty account. That resulted in alerts being sent to the Slack room and it also enabled two-way integration. Meaning, we were able to acknowledge, resolve, and add notes to our incidents directly from within Slack. It was so effective that I did not even need to have the PagerDuty app installed on my phone.

The Teams “integration” on the other hand, just provides an incoming one-way webhook that fires and forgets. Which is totally useless. We don’t need yet another way for PagerDuty to alert us of an incident. PagerDuty’s entire reason for existing is to alert the hell out of any device on the planet and it is really good at it. What Slack gave us was the ability to manage our incidents within Slack. Teams does not.

Presence indicators…don’t

There are two problems with the presence in Teams. For background, the Teams nomenclature is a little redundant and confusing. The app, MS Teams, has teams in it. But teams are just containers that you cannot interact with. Within a team are channels. Channels are analogous to rooms in Slack parlance, and it is where the action happens. Users in a team are not necessarily active in every channel within that team. Got that? OK.

First problem: there is no practical way to know which team members are active in any given channel. It is possible to see which users are in a team, but since a team is just a container, that is not useful information. It’s unbelievable that I can’t see which of my co-workers are in the channel with me. This is such basic functionality that I don’t even know how to begin to explain how fundamental it is to online communication.

This next issue may be a bug. At least, I hope it is a bug because I am not sure what rationale there could be for intentionally developing an inconsistent experience such as the one I am about to explain. The presence indicators on the desktop app allow me to select that I am “Available”, “Busy”, “Do Not Disturb”, “Be Right Back”, and “Appear Away”.

For some reason, the presence indicators on the mobile app don’t allow me to “Appear Away”, I can only be “Away” (the difference is…what?). I also have the ability to be “Off Work” on the mobile app, but for some reason, I can’t be “Off Work” on my desktop. I can only “Appear Away”.

I get that this is not the end of the world, but it really highlights some crappy development decisions. Why on earth would there be two different sets of statuses for the desktop and mobile app?

Copy and Paste…doesn’t.

Copy and Paste are broken in the Linux desktop app. Perhaps in other apps too, but I don’t know as I don’t use them. Lest you think I am making this up, let me point out that Microsoft Lync has had broken copy and paste back as far as 2012. Nothing says recycled code like 7-year old bugs cropping up in your shiny new app.

Copy and paste debuted in Windows 3.1. How does it get broken decades later and stay broke for so long?

Now let’s move on to things that aren’t exactly bugs or broken code, but are inelegant and do not reflect the needs of remote workers.

Time zone indicators

This is a VERY useful feature that I used constantly in Slack. As a remote worker, my teammates are all over the globe. Slack shows the current local time for any person in their profile data when you click on the user’s avatar. That is very useful when determining whether to ping someone at that time or to sort out a common meeting time.

No such luck in Teams. It is meant for people all huddled together in one office so nuances like time zones don’t hit the radar.

My own chat room

OK, I admit this is a little bourgeoisie of me. Slack gives everyone their own private chat room. This is useful for testing integrations (ACTUAL integrations) and as a cross-platform clipboard for info you want to move from your desktop to your phone, etc. It’s another really useful feature that only a company that understands remote work would come up with.

Teams has no such thing so I have to resort to creating my own team with myself as the only member.

Some final griping …

Bugs aside, the main reason that Microsoft Teams is so poor choice compared to Slack is that I am a remote worker. Office workers don’t really have a great need for a chat client. Office workers definitely use chat clients, all day long, but if it blows up or doesn’t work well, they just walk over to their co-worker’s desk and life goes on. The impact of a broken chat client in a traditional office is minimal and the traditional office is Microsoft’s target user base. It sells gigantic on-prem services to big companies that occupy entire floors of office space in buildings. There’s no reason to address functionality aimed at remote workers, especially when you can afford to give away the mediocre product for free to drive your competitor into the ground.

I work for a remote business unit inside a non-remote company. It’s easy for our office-bound management to make the financial choice to switch to Teams because they do not use Slack in the ways that remote workers do. If I were to sit at a desk in an office and type Hello World into Teams and Slack, I would not be able to discern any real difference between the two and also assume they were adequately matched, but one is free. The truth is that real work is done in the cracks, in the hidden nuances of the product, and that takes hands-on experience to discover; experience that these decision-makers don’t have. The good news is that the management of my company listened to the heavy Slack users within the company and are turning back to Slack after a few months of pushing Teams. That decision surprised and delighted me, not just because it improves my day-to-day, but also because it signals to me that remote workers have a voice.

As I mentioned earlier, Slack is not blameless in this. Aside from the spectacular lack of foresight demonstrated by provoking Microsoft, Slack has enjoyed very, very expensive licensing fees. Even if Microsoft did not give Teams away for free, it could still afford to vastly undercut Slack’s pricing model. I don’t know how Slack missed the fact that Microsoft could eat its lunch whenever the mood struck. Yes, remote workers such as myself really prefer Slack but despite what you hear, remote work is still a small sliver of the workforce and we don’t generally have a large voice or vote.

my shorter content on the fediverse: https://cosocial.ca/@jonw

I’ve been playing with Expect lately. Expect is an extension of the TCL scripting language developed in the 1990s. Its main purpose in life is to automate terminal interactions and it does that job very well.

I spend most of my day in a shell and automate as much as humanly possible so that I can be as lazy as humanly possible. Using tools like ssh and scp it’s very easy to automate simple commands and simple file transfers. But when these tasks become complex enough that they need to respond to terminal prompts, or provide arbitrary changing input, those tools fall apart.

My particular use case was a need to grep through logs on multiple Linux servers. This would be a trivial task to achieve using plain old ssh except for the fact that I use a Yubi key to log on to the servers. I need to interactively provide the PIN for my Yubi at each login. The same problem exists for encrypted public keys. For a while I just copied the PIN and pasted it at every prompt, but that became a pain pretty quickly so I started casting around for other options.

There’s a few solutions that specifically address the ssh password problem such as sshpass and ssh-agent, but I wanted something more generic that I could continue to use once logged in to do complex things. Enter expect.

Expect can handle some very complex interactions, but at its core is a very simple game of catch. Tell your script to “expect” some output and then “send” the appropriate input. Here’s a very simple example of SSH-into a server with a password.

#!/usr/bin/expect -f set host [lindex $argv 1] set password "password" spawn ssh "$host" expect "password: " send "$password\r" interact

You can chmod this script to make it executable chmod +x script and run it using the format ./script host.com

The interact command is important. It tells expect to release the terminal so you can type things and do things. Without the interact command, you won’t be able to…well…interact. If you’re just planning on executing a lot of commands without any interaction that is fine.

This little snippet is meant as an example and as such it stores the password in the script. Obviously, that’s crap security and you should not do that, but what options are there? One option is to store the credential in a separate file that does not comprise the codebase. Expect can read external files using this format (assuming a file named PASS with just one single line it, your password):

set fp [open "PASS" r] set PASS [read $fp]

This will store the contents of the file PASS in the variable $PASS. You can then replace the send "$password\r" with send "$PASS\r".

Another useful feature is Expect’s logging function. In my case, I am running a bunch of greps on log files and I want to save the output locally to my machine so I can review it later. The expect command log_file works perfectly for this.

Using the command log_file by itself will write a log file into the current directory on your local machine. You can also specify a log file using the format log_file output.log

To start logging, you simply put the log_file command in the script where you'd like the logging to start. If you want to end logging at some point, put log_file by itself again (regardless of whether you specified at log file name or not when you began logging). Expect is smart enough to know it already has a log file open and knows to close it when it encounters a single log_file all by itself.

Let’s put this all together to log into a machine, grep the Apache log file for something, and then save the log for review.

#!/usr/bin/expect -f set host [lindex $argv 1] set fp [open "PASS" r] set PASS [read $fp] spawn ssh "$host" expect "password: " send "$PASS\r" expect "$ " log_file website_logins.log send "grep login /var/log/httpd/access.log\r" log_file

Now run ./script host.com and sit back. In a few seconds you should have a file named website_logins.log in the script directory that contains the output of your grep.

There’s a lot more to learn, but these basic building blocks will get you started.

my shorter content on the fediverse: https://cosocial.ca/@jonw

Repetitive Stress Injuries (RSIs) were almost unheard of until the 20th century. I am sure some repetitive tasks caused them, but work wasn’t the all-consuming thing it is now and we did not have entire clinics and organizational units devoted to keeping us perpetually in that state of almost pain that our work lives create. People had more active lives before cars and computers shrunk the world. They had a thing called balance before we started cramming everyone into cubicles and rewarding those who worked longer than necessary hours. RSIs are created by the crappy work life that most of us have to endure in much the same way gyms are a self-licking ice cream cone. If we had healthy lives, we would not need gyms. If we have healthy lives, we would not get injuries from doing silly things like pushing a mouse back and forth all day.

My particular injuries are usually novel – I don’t normally get the exact same RSI repeatedly, but there are general categories that my injuries fall into. Eyes, arms, back and sometimes feet. OK, ok…that is pretty much my whole body.

Eyes

The most distressing category is my eyes. Humans tend to blink less when they’re staring at a screen. That’s not a problem for most folks because your eyes can recover after a slow-blinking movie just fine. But for those of us that stare at screens all day long, it takes a toll on our eyes. Many office workers have dry eyes because years of unnaturally slow blinking causes the oil glands in our eyelids to decrease function which is the primary way our eyes are lubricated. In more serious cases, the eyes can become so dry that calcium deposits can grow under eyelids which aggravate the problem even more because they scrape the surface of the eye at every blink.

Treatments for dry eyes range from hot compresses daily to re-open the oil glands, all the way up to minor surgery to remove the calcium buildups. Eye drops are a temporary measure but generally can’t help long-term. So, keep hydrated, use the compress, and hope for the best. That’s all I got for eye treatment.

Arms

I think we’re all familiar with forearm and wrist/hand RSIs by now. The most widely publicized is the old “carpal tunnel syndrome” which is definitely painful and can be hard to fix but is not the most common RSI. Personally, I get aches and pains in my lower forearms and, believe it or not, the base of my index fingers. Weird, right? I get this condition where rotating my hand, such as I’d do to twist a doorknob, will send shooting pains up my index fingers. I am pretty sure that one comes from the way I hold my mobile phone, which I use constantly during the day.

Treatment for those types of RSIs is to simply move things around on my desk and use my phone less. I have two mice on hand; a typical optical mouse and a very atypical “vertical mouse”. The vertical mouse took a lot of getting used to, but because my arm is held in a completely different way than a normal mouse, switching and forth between them works very well to fend off the old RSI.

A great way for me to use my phone less is to uninstall Slack. The vast majority of my work use on my phone is Slack and my removing the app I end up using my phone hours less each day. If you take a look at your screen time app (depending on what type of phone you have), you may be surprised at how much time you spend on work apps. Removing a work app, even for a few days, can dramatically change your habits and give your hands a break. This technique can work for other heavily-used apps as well, depending on your work situation.

Back

The back is a tricky beast and I have had a long battle with mine. I injured my back about 10 years ago and my doctor at the time was a “walk it off” type of guy so I did not receive adequate medical treatment for it to heal properly. Because of that, I have some scar tissue where the injury occurred and because scar tissue does not stretch as nicely as muscle or skin, my back is always tight and I have a heightened potential for re-injury. I am extremely cognizant of this and take lots of precautions which is why I can still lift and install 90lb servers in data centers by myself despite this injury.

Treatment for my back involves a few things. First, I have a standing desk that my employer provided for me (thanks!). This allows me to raise and lower the desk to different positions during the day. For meetings I generally stand, for coding, I generally lower it and sit, but the point is that I can do anything I want. Neither sitting nor standing is a cure for ergonomic injuries, but changing positions definitely is. I also have two different chairs – a “standard” office chair and a wooden stool. I switch between them frequently and because each chair forces me to sit in a different posture, that also helps keep the RSIs at bay.

Feet

I was surprised when I started getting RSIs in my feet. Like, who has ever heard of such a thing? We all know about foot ailments like plantar fasciitis, but this is not that. This is a pain in my feet that I finally figured out stems from a uniquely work-from-home scenario: I don’t generally wear shoes in the house. Not a problem for regular folks, but when you work from home that translates into “I almost never wear shoes” because I am not commuting to an office for work. That can cause problems with your feet if you need the support that shoes provide. I now have a set of designated indoor sneakers that I wear and my foot pain is gone.

We’re fixing the forest, not the trees

Current ergonomic workplace practices seek to fix the symptoms, but not the root problem itself. Your office purchased a bunch of work stations that are suitable for statistically average people. Most people are this height with this length of arms, therefore you need a hundred of this chair and this desk. When employees find that these tools are not working for them, the ergo folks start bolting stuff onto that work station to try to make it fit. They add ergonomic mouse pads and keyboards, specially adjustable chairs, and monitor arms to better position the screens. Sometimes that works, but those are all kludges that typically just make the problem bearable instead of actually fixing it.

The reason that workplace ergonomic issues are hard to solve is that the entire office work experience is a made-up fantasy that is fixated on the age-old desk. Everything at work surrounds your desk so every solution is an attempt to fix the desk. A true solution would get rid of the desk. Well, I don’t mean that literally – we all need some hard surface to put our stuff on, but de-emphasizing the desk is where I am going with this. If the desk wasn’t so central to everything, people can do what I do: move around freely, work in a variety of positions, and generally make my work life reflect a healthy, normal life that includes constant movement.

I’ve found that the only long-term solution to workplace RSIs is frequent movement. By changing position and moving around, you eliminate the “R” (repetitive) in RSI because simple changes in position change the parts of your body doing the work. That is my takeaway for you – move around more and you’ll feel better.

my shorter content on the fediverse: https://cosocial.ca/@jonw

image courtesy of Arista

There are some things in a sysadmin’s life that simply need to scale. If you’re working in a growing footprint, then switch wrangling is one of those things. I reached that point at about a dozen switches in 6 different locations but just got around to building something recently. We use Arista switches and they have a built in XMPP client which made this really easy.

The most well known implementation of XMPP is probably Jabber. Jabber is an instant messaging client/server system that operates using the XMPP protocol. It’s cross-platform — it probably has more clients and servers than any other IM system in existence, and as such it’s pretty easy to find the parts to get this running.

You’ll need three things:

1. The switches need to use an XMPP client. Good news, this is built in to the Arista EOS.

2. Switch admins will also need to use an XMPP client to talk to the switches. Good news, there are a billion Jabber clients for every imaginable platform.

3. You will need a Jabber server for you and your servers to connect so you can ‘chat’. Good news, there are a billion Jabber server for every imaginable platform.

I am not going to cover installing and configuring an XMPP server. That documentation has been done to death on the internet so you should be able to grab an XMPP server for your platform here, and a client for your platform here.

If your new to the Jabber server thing I recommend using ejabberd. It is mature, well documented, and there’s a ton of information about it around the web to help you out.

Once you have your Jabber client talking to your Jabber server, come back and we’ll set about configuring the Arista to enter the fray.

Switch XMPP config

The Arista documentation for XMPP is in the Session Management section of the applicable EOS version manual. The basic set up requires you to tell your Arista what XMPP server to use, and some credentials to log into it. A more advanced setup allows your Arista to join group chats which is an extremely powerful feature.

To get the basic client working, you need to enter config t mode in your Arista which may require an enable password depending on how your Arista is set up.

enable config t management xmpp

Your prompt should now be (config-mgmt-xmpp)#.

Enter the following commands, substituting the variables with real info that matches your setup:

shut server $YOURJABBERSERVERFQDNORIP username $ARBITRARYNAMEOFTHISSWITCH password 0 $PASSWORD session privilege $PRIVILEGELEVEL domain $YOURXMPPDOMAIN

Some notes on the variables:

1. $YOURJABBERSERVERFQDNOR_IP: Pretty self-explanatory. Some routable domain or IP address that points to your Jabber server

2. $ARBITRARYNAMEOFTHISSWITCH: Make it something you will recognize in chat. Something like LA3-unicast is a good name.

3. $PASSWORD: There is a number preceding this (in this case 0) which tells the switch if the password is encrypted. If you’re configuring a pre-encrypted password use password 7 $PASSWORD instead. More details are in the Arista Session Management link.

4. $PRIVILEGE_LEVEL: This is the default access level commands coming through XMPP will inherit. It needs to be high if you want to be able to actually make configuration changes on the switch. Much lower if you only want to query the switch in read-only mode such as checking version numbers, etc. Look at the PRIVILEGE EXEC section of the Arista Session Management link for more info.

5. $YOURXMPPDOMAIN: this is just a namespace, not a FQDN. You will have already set this in your XMPP server config, so use the same domain here.

You can now issue the no shut command which should make your switch join your Jabber server. You won't notice this at first because your switch is not one of your contacts yet. Add the switch as a contact in your Jabber client and then you'll see it show up in your contact list and you can now chat to it like you would a human.

Jabber client uses

Single switches

The most obvious thing to do is talk to your switch. You can issue read-only commands like show version or sh arp or show int eth1-8 counter rates and get responses back from the switch.

If you’ve configured a higher privelege level, you can also make configuration changes to your switch such as add BGP configs.

It may seem like this isn’t all that much quicker/useful than logging in to your switch via shell, but it probably is. Consider that these commands are issued through a Jabber client — any Jabber client — even a mobile Jabber client. This makes accessing your switch possible from pretty much anywhere at any time.

Switch groups

In my opinion, the real power is multi-user chats. Or, more correctly, multi-switch chats. You can tell your switches to join multiple group chats by adding as many as the following lines as you’d like:

switch-group all-switches password 0 $PASSWORD switch-group anycast-switches password 0 $PASSWORD

This switch will now join two chat rooms; one for all switches and one for only anycast switches. These are just examples, but you are hopefully starting to get the idea of how powerful this is.

Note: You need to create the multi-user chat rooms in advance on your ejabberd server. I assume other XMPP servers require the same pre-configuration. Having all all-switch group chat allows me to issue a single command and get a response back from all my switches. In the example below I simply ask for the EOS version, but you can easily ask for throughput data or mlag status or anything else that you think is interesting.

Issuing multi-line commands

This stumped me for a while, but eventually I made sense of the single blog post on the internet that casually mentions a solution. Because you’re communicating with your switch via an XMPP client, the switch does not preserve state. This means that you can’t put the switch into config mode with one message and then proceed to issue it configuration commands in subsequent messages. As soon as you put it into config mode, it pops back out. This makes issuing multi-line commands difficult.

I assume this is a security feature to prevent switches from lying around in configuration mode, but I am not sure.

The answer is that you need to send all your commands in a single chat message. Take a look at the second screenshot in this post. I issue all these commands in a single message:

enable config t router bgp 12345 network 192.168.1.1/24

This adds the nonsense BGP ASN to my switch in one command.

Security

Arista XMPP support is an incredibly useful feature, but I cannot stress enough that accessing your switches via XMPP creates a huge security issue.

From a server security perspective it is not too bad. The XMPP support in the Arista is to act as a client so you are not opening up any ports on the Arista. However, the very thing that makes this so convenient is the very thing that makes it such a big security problem.

Consider that anyone who can log in to your XMPP server would also be able to add your switches as contacts. Another consideration is losing your phone/laptop/tablet that has a logged in Jabber session with all your switches.

To mitigate these security issues, consider at least the following:

1. Use the lowest privilege level setting possible in your Arista config. There’s an argument to be made about only allowing read-only access via XMPP. That would allow a sysadmin to keep an easy eye on traffic and do basic troubleshooting such as arp issues, but not allow any configuation changes to be made via the client.

2. Always stand up your own XMPP server. It’s theoretically possible for your Arista to join any XMPP server, even a public one, but that would obviously be an insane thing to do.

3. Lock down your XMPP server using the same best practices you would for your other critical machines: least privilege, firewall, OSSEC or other HIDS, 2FA if supported, etc.

4. Consider a company policy that disallows mobile clients connecting to the XMPP server to mitigate lost phones, etc.

That’s it. Enjoy your new chat buddies.

my shorter content on the fediverse: https://cosocial.ca/@jonw

I recently had a need for a link shortener and that simple need quickly turned into an idea to build one from scratch. I figured it can’t be that hard, and I was right. Shortening the links is easy, but dealing with the unwashed masses of the internet that are going to use it is another problem entirely. I have lots of little internet-based projects that are locked down for only my use because the internet is rife with people who just want to watch the world burn, and they’ll try to destroy everything that comes across their path. I met some of them during my first week with Link Chomp.

The mechanics

I went into this project with my eyes wide open to infosec concerns. I maintain infrastructure for an infosec company and while I am not a researcher, I have a healthy exposure to the types of bad things people do on the internet. Based on that experience, I designed the architecture with these concepts in mind:

1. No databases. Everything I install presents yet another attack vector. There are some things that I simply need, such as a web server, but database servers are a really big target so I said “no” to that.

2. Segregated into micro-services. Historically, most web apps are monolithic hunks of code and a change to one area of it ripples through the rest of the app. Modern thinking eschews that idea and instead embraces the concept of micro-services. Under a micro-services architecture, an app is not one thing. It is comprised of smaller services that run independently of each other. This allows greater availability because if one service has a problem, the others keep humming along. It also allows for better security because if one service is compromised, that does not open the door wide for the entire application to be compromised.

3. Try to make the links safe. I’m tilting at windmills with this concept, but I need to try. Link shorteners ostensibly just…well…shorten links. So a long link like https://www.pluralsight.com/courses/aws-certified-cloud-practitioner becomes a short little guy like this: https://cmp.cx/8a971. Keen readers will notice that the destination link is now totally obscured. Bad guys know that and they use link shorteners to trick people into clicking bad links by hiding the true destination. I try to address that.

More detail

Let’s examine each of those points in more detail.

No database

I have two concerns with running a database. I already mentioned the first which is how attractive databases are to attackers. They’re attractive because they’re complicated and it can be hard to write code that prevents really clever attackers from extracting data from the databases. Virtually every data breach we hear about these days takes the form of a bad guy successfully exfiltrating data from a database.

The other main reason that databases are attractive is that bad guys can sometimes put data into a database, not just take it out. A bad guy can write a piece of code to…say…email the contents of the login form to them and then store that piece of code in the database so it executes every time someone logs in.

Link Chomp avoids this by devolving to a much earlier technology: text files. While it is certainly possible to exfiltrate or inject data into a text file, it is harder. It is harder because text files are easier to secure using basic Linux file system attributes and simple is always better.

There are downsides to this decision. The main one being concurrency. Database servers know what to do if two Link Chomp users create a new “chomp” (my cute name for a shortened link) at the same time. The database server can make sure that two people don’t accidentally get the same chomp code, or don’t accidentally overwrite each other’s new chomp whereas I have to do all that work myself because I use text files.

Micro-services

I will be the first to admit that I could have pursued this a little farther than I did. I broke Link Chomp into three services: the interface where people go to create new chomps, the service that redirects users when they click a chomped link, and the background services such as performing backups and expiring old links.

The process of creating a chomp is quite involved – there are many steps such as ensuring the destination domain looks properly formatted, checking that the link is not outlandishly obviously malicious, ensuring there are no duplicate chomp codes, and then the whole process of recording all this stuff. I could have broken all those into separate services, but because the application requires almost all of that to happen, a single broken service would break the app anyhow, so the risk/reward ratio was not in favour of doing the extra work.

However, the chomp code is very portable. It has no dependencies outside of a few standard PHP modules and the code flow is broken nicely into compartmentalized functions. That makes it very easy to troubleshoot problems and also makes it easy to add new functionality. For example, I am always messing around in the function that checks if a link seems bad. It is a work in progress that likely will never be very accurate, but I am always adding tests, then evaluating if those tests impact the speed of the site, and then adjusting as needed.

Safety

The last paragraph above sums up the safety issue. Because we know bad guys use link shorteners to obfuscate destination URLs, I am compelled to make an effort to prevent that. But, let’s get real, that is pretty much impossible. Bad guys don’t link to obvious things like www . thisisabadlink . com. They know that’s going to get banned everywhere. Instead, they spend a ton of time breaking into legitimate websites and putting phishing pages on those sites. Because of that, bad pages that contain phishing code, or credit card stealing code, etc. usually reside on legitimate websites that are not on any blacklists and are very hard to detect as being bad simply from the link. In theory, it would be possible to scan the page looking for bad code, but that is simply too slow and would make Link Chomp unusable.

So, what do I do instead? I do some checking against a bad word list and I check to make sure the domain is properly formatted. I also cap the URL limit which is kind of a weird decision. Link Chomp is a link shortener so it is assumed that users are going to come to it with really long links. But before I capped it, I would routinely see internet goofs pasting in URLs thousands of characters long. So, capping is required and there will probably be a few legitimate users caught up in that, but I don’t have a better solution right now.

Next, Link Chomp resides behind the Sucuri firewall (this is not a secret, anyone can see that if they know how). Sucuri has a nice API and I use it to block repeated bad requests. If someone is obviously hammering Link Chomp with bad URLs, I issue an API call to Sucuri to block their IP. IP-based blocking isn’t perfect, but these guys are not determined, attackers. They’re just being knobs and they go away once they encounter even a small bit of resistance like this.

Future plans

Now that the framework is built, it is easier to add functionality. Some of the ideas on my road map are:

URL blacklist checking

I’d like to check submitted URLs against a blacklist. The obvious choice here is Safe Browsing, but there are a few others that I am considering. This is a very important decision because the check has to be extremely fast. I do not want to add a second or two to the chomp creation while we check a blacklist.

Initially, I thought that using a service that would let me download a list of blocked URLs would be best because I can check that very quickly on the server. However, I quickly realized that idea was foolish for a few reasons.

The first reason that idea will not work is the sheer size of a URL blacklist. There are just millions upon millions of bad sites out there and I don’t think it is feasible to handle a file that large at run time.

The second reason is that the list would always be somewhat out of date. I would download it periodically, but probably only daily, or maybe a few times a day. The problem with outdated blacklists is that the bad guys are putting the most effort into spreading their bad link via phishing emails in the first few hours after they create it. Some percentage of the users receiving those phishing emails will report it, and the URL will be blacklisted reasonably soon. If I am using even a slightly outdated blacklist, Link Chomp will be blind to the blacklisted domain for longer than it would be if I were checking in real-time.

Why do I care? Well, when you chomp a link you get a shortened URL back from the cmp.cx domain. That means there are tons of links our there using cmp.cx and I do not want that to get blacklisted. If cmp.cx were to be blacklisted, it would cause a lot of problems for the people using those chomps.

So, there is work to be done here. I am not sure what the final solution looks like yet.

Custom links and subdomains

I started this project because I wanted subdomain support which, ironically, I have not built in yet, but I think it is a great idea. Subdomains are domains tacked on the front of a domain, for example jonwatson.substack.com - “jonwatson” is a subdomain of “substack.com”.

Subdomain support would allow shortened URLs in the form of 8a971.cmp.cx. What you get back from Link Chomp now is something like cmp.cx/8a971. The reason I want that is because I want to deploy a wildcard cert for *.cmp.cx and then I can offer TLS secured subdomain forwarding. Maybe nobody cares about that other than me, but I like the idea.

Custom links are a similar, but different thing, that allows users to specify the chomp code part of the URL instead of accepting the randomly generated one. For example, this random chomp https://cmp.cx/4907b can be set to https://cmp.cx/notarickroll. Ok, it totally is a Rick Roll, but you get my point.

The blocker with these two ideas is that I want custom stuff like this to be a premium service, and not open to the unwashed masses. To support that, I need to support accounts – the ability for users to make accounts, and that has a whole chunk of work behind it so that is slated for some future weekend.

Final thoughts

Every project I’ve worked on is always more complex than it seems. I did not know exactly what the complexities would turn out to be with Link Chomp, but it did not disappoint. However, the basic idea is simple so the complexities did not become insurmountable over overwhelm me. And now I am in a position to build smaller, neater, features into it as I go along.

You can find Link Chomp here.

my shorter content on the fediverse: https://cosocial.ca/@jonw

Back in January, I decided to take my writing up a notch. I used to just “blog” which was this nice relaxed pace, writing only when I felt like I had something interesting to say. While that provided almost no stress, it also didn’t provide much content. It wasn’t that I felt I had nothing to say, I just felt like someone had already said it. I had no incentive to write about the things I like because a million other people were already writing about those topics. At around the same time, I started to become actively aware of the decline of quality in the mainstream media.

Google has almost singlehandedly destroyed the news market by creating Goggle News which essentially denies the ability of any news outlet to make money on their product. The news media responded in the same way that every other industry that has been butchered by a Google or an Amazon can, and either shut down or started producing the only content it could afford – poorly researched and hastily written articles. Eventually, I came to realize that I hadn’t read a “mainstream” newspaper or watched a “newscast” in months, perhaps years, preferring to get my information from individuals and smaller niche sites that didn’t need Google to get by and were able to survive, if not exactly thrive.

Yesterday’s “fringe” content dominates today’s conversations.

I came to realize that these two things naturally came together. Of course there were a million people all saying the same thing! The vacuum left by Google’s outright attack on the fourth estate sucked these voices in from the fringes and they became prominent. It was at that point that I decided to write more. Write stuff I knew about. Write it in my own voice instead of conforming to the dull, lifeless drone that media companies had spent decades honing to perfection. Write the stuff I want to write without second-guessing myself about tone; without trying to write “evergreen” content for the clicks.

I (perhaps naively) believe that I create useful content that people like.

To kick this off, I joined Medium and for several weeks I wrote articles there. It did not take very long for me to realize that although my idea was good, my solution was not. Medium is a very broken idea, but you need to really muckle on to it and live it hard for a little while to discover that. My final article on Medium was about how it had unwittingly built a system that rewarded writers that published poor quality articles while it was attempting to do the exact opposite.

I am a technologist at heart. I build things every day. Some are simple. Some are mind-blowingly complex. Some are embarrassingly stupid and will never see the light of a Github repo, and some are the pride and joy of my career. Because of my long experience building things, I know that the first version of anything never works. It is always a learning tool and will always be thrown out, so I looked for other solutions.

“Plan to throw one away; you will, anyhow.”
– Fred Brooks, The Mythical Man-Month

I reflected on the fact that I missed mainstream media so little that I did not initially realize that I stopped consuming it. It was only after being absent from it for months, perhaps years, that I said “heeey…I wonder if the Globe & Mail newspaper still has door delivery to my little town”. (Spoiler; it no longer does and the digital edition is now the same price as the paper edition that used to be delivered to my door.)

I wondered: if I am no longer consuming traditional media, am I ill-informed? Do I not know what is happening in the world anymore because I stopped reading newspapers and dropped my subscriptions to news outlets like CBC? I was careful to eliminate bias confirmation as I explored this idea. I purposely reviewed the media outlets that I had let slide into oblivion to see what they had been reporting on and whether I knew about it already. And, if not, why not?

My investigation ended with the confirmation that I was, indeed, informed. And while I certainly encountered stories I had not heard of, they were not stories that would have interested me had I known. Things like changes in hunting seasons and local stuff in other provinces that have no effect on me are some examples. I came to understand that I did not check out of reality, I just subconsciously made choices about outlets that I considered to have quality information. It’s no surprise that the suffering media outlets I grew up with did not top my list of purveyors of quality news.

Aggregated news is a product, not a legitimate attempt to inform.

Instead, I realized that I now get my info from specific sources, and frequently collaborated more complex issues with other resources. Instead of paying the Globe & Mail for its Ontario edition – which is the closest edition to my east-coast home, but still several provinces of relevance away – I pay for a monthly subscription to All Nova Scotia.

In lieu of hoping that the doddering CBC or CTV or will get even basic information security news correct, I read people like Brian Krebbs and follow sites like Threat Post and Dark Reading.

Rather than relying on a third party to curate information that is in its best interests to feed me, I started going to the source. I check-in with my town website to know what is important hyper-locally. I watch the daily COVID-19 briefing from our provincial Premier and Chief Medical Officer of Health to see what the health orders and epidemiology says about my near future. I reach out to my Mayor or councillor directly when I have questions about things that fall within their domain. And when I need more than a three-bullet PowerPoint slide to understand complex legal issues in the digital landscape in my country, I go to Micheal Geist.

Opinions aren’t news

I now eschew opinion pieces in a long-in-coming epiphany that o pinions aren’t news. There is no value other than a temporary fanciful distraction in knowing what doctors think about legal matters. There is no value in paying attention to what economists think about epidemiology. And there is no value in news outlets that have lapsed into hapless opinion mills because that is the only unique content they are able to muster anymore.

It is with the sincerest of hopes that I can contribute meaningful content and commentary to the discourse that I now decide to reel in all my writing activities and concentrate on one single project: my Death By Tech newsletter/blog that you’re reading or listening to now. I will no longer be continuing the One Time Pad security newsletter and I will be removing all my content from my main site at jonwatson.ca and focus solely on writing content here.

How can you support me?

Subscribe to my content, even at the free level! That encourages me and makes me want to continue.

If you’d like to send a little more love my way, you can do that too and I will reciprocate by delivering twice the content to you. Everyone who subscribes, whether for free or with a paid subscription, gets at least one post a week from me. Those who support me financially get twice as many posts.

I don’t write “better” content for subscribers. I don’t save “the good stuff” for paying subscribers. Paying subscribers just get more content because I alternate between posts. Every second post goes only to paying subscribers.

In addition, regardless of when you jump on board as a paying subscriber, I will unlock all of my previous paid content in my archive for you so you will have instant access to a ton of new content. If you’re not a paying subscriber now and you visit that link, you will see some content with a padlock icon next to the date – that’s the extra stuff that is waiting for you.

I get that we all have subscription fatigue. $5 here and there is a pain to manage and it adds up. To help with that, I am offering many, many ways to contribute – you probably already have an account with one or more of these services that you can use.

Here's some actionable information:

If you are a free subscriber:

• You’re helping me already. Thank you! Would you consider becoming a paying subscriber to support me even more? You can use this link to get 50% off if you subscribe directly here on SubStack.

• If you would prefer to support me via LiberaPay, PayPal, BAT, or Crypto Currencies, please click this link for info on how to do that.

• ALWAYS share the free editions you receive with anyone you’d like.

If you're already a paying subscriber:

You’re already supporting me. Thank you! If you want to help even further, you can:

• Send this two-week free trial link to anyone you think may be interested: https://jonwatson.substack.com/2weeksFree

• OCCASIONALLY share the paid editions you receive with a friend or two that you think will like it.

If you're not a subscriber at all yet:

Maybe you stumbled across this post or perhaps you arrived here via an article someone shared with you. In any case, thank you for reading this far. If you’d like to support me, the best way to do that is to become a subscriber, either free or paid.

• If you’re not sure you want to take the leap right away, subscribe for free to get at least one edition per week. To subscribe, click that big blue button at the top of this page.

• If you’re on the fence about becoming a paying subscriber, feel free to use this link to get a free two-week trial during which you will get both paid and free editions. That is at least two editions per week.

• If you would prefer to support me via LiberaPay, PayPal, BAT, or Crypto Currencies, please click this link for info on how to do that.

Regardless of what you decide, I hope you came away from this post with a better sense of what I am trying to accomplish and how much I like creating content. Thank you for reading.

my shorter content on the fediverse: https://cosocial.ca/@jonw

Data Centres are made for servers, not humans. Consequently, they are inhospitable places and prolonged exposure to this adverse environment can quickly take a toll on your productivity and your health. Once your health starts to go, your attitude and your deep-thinking abilities go with it and the quality of your work drops. You owe it to yourself and your team to remain as effective as possible while onsite, and here’s some tips to help. I’ve spent a few weeks in several different data centres around the globe this year, and here are some things I’ve learned that can help you out if your destined for one of these hell holes.

Make sure you can get in

Data Centres (DC) are very attractive targets to bad actors. DC operators know this and generally employ pretty stringent identification measures before granting access. Large companies also usually have the clout to enforce additional identification measures on top of what the DC requires, so you need to be prepared to prove yourself to get in. It’s not uncommon for the foyer in a DC to be filled with people on their cell phones calling back to the home office for some kind of authorization they need and if their home office is 15 hours time difference from the DC, they ain’t getting in today.

You should have some kind of customer service representative at the DC. Contact him or her prior to your visit to ensure you know what you need to get access. If you don’t know who your rep is, call your account manager (your billing department should know who that is) or call security directly at the DC and ask about their procedures. As an example, the access process I’ve most commonly seen is:

• A site visit ticket has to be filed prior to the site visit. It lists the full names of everyone who is coming, and what days they will be there.

• Passport is required onsite to prove you are who you say you are.

• Some companies also require you to show your company ID to the DC.

Also worth noting is that some cages have PIN locks on them. Even if you get through security and up to your cage, if you don’t know the PIN to open it, you’re not going to be able to do much. If I am going to a DC I have not been to before, I ask the DC techs to take a picture of the cage, closed and open, front and back, and send it to me. That way I have a good idea of the lay of the land before I get there.

You can’t stay there from dawn to dusk.

Well, you CAN, but you want to break it out into a few hours at a time with breaks in between. If possible, I would not spend more than 3 hours in a DC without a 15-minute break to hydrate and give my lungs a rest. You’re going to be exposed to non-stop blasting dry air, alternating between very cold (in the cold aisle) and warm (in the hot aisle). If you’re installing equipment, you’re going to be moving between the hot and cold aisles constantly which means your ambient temperature is going to be bouncing all over the thermometer as much as 20 degrees (Celsius) between aisles. That is a ripe condition for getting a cold.

If you’re doing maintenance that doesn’t require both sides of the cage, then you’re going to be exposed to either too hot, or too cold air for the duration of your stay. The air is also dry as dry can be. You’ll dehydrate, your throat will hurt, and because you’re not allowed any liquids on the floors, you’re not going to be able to stay hydrated on the floor. Every DC is different — some have break-out rooms on each floor where you can quickly leave your cage and spend some time in a more hospitable environment periodically. Some DCs are less convenient and have no facilities to sit down anywhere but on the main floor so unless you want to crouch in the hallway for your break, you’ll need longer breaks to get down to the main rest area.

Think about power

In most cases, your servers are not going to have the same power couplings as your laptop. And, in most cases, you’re going to need your laptop to help with whatever task you’re doing. Therefore, a dead laptop can end your day really quickly. North American servers usually use C13/C14 plugs pictured below. C13/C14 plugs

C13/C14 power cable Most laptops do not have this type of power cable and while you would think it would be obvious for DCs to scatter regular NEMA 5 plugs around the walls, they don’t.

NEMA 5 You’ll decide what plugs your Power Distribution Units (PDUs) have in your cage, but I generally use the C13/C14 and an adapter like this can help prevent the dead laptop battery problem.

NEMA 5/C13 adapter You’ll need to adapt this information for the power in your particular DC, but with a little foresight you should be able to ensure you can power your laptop onsite.

There are no chairs or desks (aka — cable length matters)

I haven’t seen every DC in the world by far, but the one constant between all of the ones I’ve seen is that there are no chairs or desks anywhere on the floors. Therefore, you’re going to be spending a lot of time sitting on the floor. If you’re not a millennial, this will suck. You will very quickly get tired of getting up and down repeatedly but other than lugging a chair in with you, I don’t have any solution for that. However, one important factor that can help out a lot is cable length. If you have to hook into a device at the top of your 42-U rack and need to sit on the floor to give your laptop something to rest on, then you will need a 6-foot cable at a minimum, and 8-foot is better. When shopping for cables, always get long ones. There’s nothing worse than having to spend hours in some contorted position on a DC floor because your serial cable is too short to sit comfortably.

In theory, you should be able to slide one of your servers out enough to make a little desk for your laptop. In practice, that rarely works because the cabling on the back of that server prevents you from sliding it out far enough. DCs also have “crash carts” which are mobile monitor and keyboard setups. You can drag them to your cabinet and use them to plug into the video and keyboard of your servers to get them set up locally. Some crash carts are actual carts and have spare room on them where you can lay your laptop. But most I’ve seen look like this and have no extra space to lay a laptop.

Fly in before, leave after

There’s a maxim in the military that goes something like this: “It’s not enough just to get the troops to the front line, they also have to be able to fight when they get there”. The basic idea here is that delivering an exhausted worker to the work site is going to result in some pretty shoddy work being done, or not being done at all. If your organization is big enough to be using a DC, it is probably also big enough to have a travel policy. And most travel policies allow for travel the day before and after the business purpose. Use it. Unless your destination DC is relatively close, don’t try to fly in on the red-eye and get to work that morning. You’ll be tired and run down and that will allow the adverse conditions of the DC to affect you even more. Fly in the day before, get a good night’s sleep, then go to work. I suppose if you want to fly out on a work day, go ahead — you can be sick at home on your own time.

I hope this helps someone out there who hasn’t been to a DC before, or is having a tough time navigating the process. Whatever job you’re being sent to do is already pretty complicated or you would just get the DC techs to do it. So, don’t cripple yourself by engineering a situation where you’re not in top form. Plan your travel, access, and breaks accordingly and you’ll be a lot more effective.

my shorter content on the fediverse: https://cosocial.ca/@jonw

Before I start into the topic I want to make a few points very clear. You are not about to read an article about why COVID-19 is not a big deal. You won’t find support for arguments such as “we’re overreacting” in this post. You won’t find sympathy for conspiracy theories that the pandemic is not real, and you won’t find a receptive ear if you think these things. Go somewhere else if you think these things. I don’t want you in my orbit.

Ok, now here we go for real this time…

What is contact tracing?

Let’s start at the beginning. I am not an epidemiologist so I may get some of this wrong. But, I have watched almost every broadcast from our Chief Medical Officer of Health and there have been a lot of those broadcasts. From them, I have gleaned some basics.

Basic #1

The primary job of the public health organization is to ensure the public health care system functions. It does this every day, but certainly, the job is much harder to do during a pandemic. Much like banks do not have enough cash on hand to allow every customer to withdrawal their balance all at once, hospitals do not have enough equipment for every citizen all at once. Public Health seeks to maintain a working level of health care based on expected demand.

Basic #2:

There are two basic pools of infected people that Public Health focuses on: those who contracted the disease in a known way, and those who contracted the disease is an unknown way. In the case of COVID-19 in my region, the first group became infected due to travel. Every early case in my province was a person who had recently traveled or had close (usually familial) contact with someone who had recently traveled. Now that the disease spread is more advanced, we have the second group to contend with which are cases not linked to any known source. That second group is called “community spread” which refers to the fact that these people got the disease from someone in their community and not someone who had traveled recently.

Basic #3

COVID-19 has an average 11.5 day incubation period. To make things easier, we’re saying that an infection cycle is two-weeks long. This means that if I contact COVID-19 today, I will likely remain asymptomatic (meaning, I have no symptoms of the disease) for 11.5 days, perhaps a little longer, perhaps a little shorter. This means that if I tested positive for COVID-19 today, I would have to go back through all my travels during the past two weeks to help Public Health determine where I may have contracted the disease. This means every shopping trip, every gas station, every dog walk, every post office visit. Everything.

Contact tracing deals with the community spread people in Basic 2, and specifically assists Public Health in tracking the origin of the disease on Basic 3.

In Basic 3, Public Health isn’t actually all that concerned about where we have been. Rather, it is most interested in who we had contact within those 14 days, but of course, those two things are inexorably linked. Because we’re all supposed to be isolating and not having people in our houses that do not live there, the only place we could have come into contact with an infected person is somewhere else. Therefore, tracking the locations a newly positive person has visited is the first step in determining who that person came into contact with. And in that pool of people is going to be one or more infected people.

Once Public Health has compiled the list of places the newly infected person has been, they then need to compare that list with everyone else who has been infected recently and hope they find a match: “they both went to the grocery store that day”, for example. However, even if they luck out and find a smoking gun like that, it does not necessarily mean those two people came within 2 meters of each other to transmit the disease. They may not have even been at the store at the same time. So it’s a good clue, but it’s not the best clue.

Lastly, Public Health will further try to classify the type of contact as Low, Moderate, or High Risk and then take different courses of action for each type of contact. Here’s an infographic explaining that process from the Nova Scotia Health Authority.

So far, I have described a lot of work. Public Health people are doing an immense amount of work to attempt to draw intersections between infected people and in most cases, the best result is a solid “probably”.

What Public Health really wants is to cut through all the guesswork and know, for certain, that you were within 2 meters of an infected person at 2:14 pm last Tuesday afternoon for more than 5 minutes at the gas station on route 4.

That is what contact tracing purports to be able to deliver. And do it by removing a lot of the drudge work Public Health is doing now.

Pros and Cons of the current contact tracing method

The biggest pro of the current method is that there are no false positives. Public Health does not get involved until someone has tested positive for COVID-19.

That may seem like an innocuous benefit because we expect things to work properly, so the fact that a process has no false positives isn’t usually considered a pro. It’s just expected. However, when you consider the massive amount of work that goes into tracking down a single infected source, eliminating false positives is paramount because there’s just not enough people and time to go chasing after false positives.

Another pro is that the current system has 100% coverage. Meaning, it is not limited to just people with smartphones.

The biggest con of the current method is that it is primarily reactive. It has very little ability to predict cases before people become symptomatic and have potentially spread the disease around.

Another con is that people just plain old forget stuff. It can be difficult to recall every single person you’ve seen or place you’ve been for the past 14 days and that can lead to unexplainable transmission.

Pros and Cons of mobile app contact tracing

The biggest pro of using a mobile app is that it can predict infections before the person becomes infectious. That means that people who have been in contact with a known infected person can be isolated prior to becoming infectious themselves and endangering others.

The biggest con of using mobile data it has the potential for a very high false-positive rate. The apps use Bluetooth to detect other devices in its area. Anyone who has used a BT headset or has BT in their car knows that BT has no problem penetrating walls, windows, car doors, and has a range far in excess of 2 meters.

Other potential cons of the mobile process include:

• It has less coverage because it excludes people who do not have smartphones, or who do not/cannot install the app or, in phase 2, do not activate it in the OS.

• It relies heavily on wide-spread testing. The relevancy of this point is highly regional. Some areas have very robust and widespread testing programs, others do not. But without broad testing, the list of known infected people will be less complete, and newly notified people will not be able to get tested quickly. A strong testing program is a lynchpin of any contact tracing, but especially rapid and automated contact tracing.

My first big idea

In my field, we have an understanding that we’re going to “throw the first one away”. Meaning that the first attempt to create a solution for a problem will likely be thrown away. This is because you’ll likely learn things you did not initially know and that will inform your way forward in a different way. This was no exception.

I initially had this super-radical idea that we should use existing data. Facebook, Google, and Apple have been collecting location data for decades on its users. Let’s make/ask them to provide this data for Public Health instead of re-inventing the wheel.

Facebook “check-ins” do precisely that – it tells Facebook where you are. Facebook then takes that location data and figures out every other Facebook user that is in that location regardless of whether those people check-in or not. And now that at least one person has checked into that location, Facebook can now record every Facebook user that goes to that place from now on. That sounded like exactly what we were looking for: “tell me everyone who was in that park at that time.”

On the slightly less seedy side of town, Apple and Google have already published maps on where people are congregating during isolation weekends. Seriously, they already have this data and have published it here and here.

Bingo. I thought this was precisely the data that we can use for contact tracing so there is no point in re-inventing the wheel. Especially re-inventing it in a way that will allow yet another third party such as a government to track all of its citizen’s movements.

However, as I researched the topic deeper, it became obvious that the existing location data is collected using cell phone towers, GPS, and wifi and that location data is not detailed enough for this purpose. Your phone knows you’re at Chik’n Chik’n Chik’n (come on, you KNOW there has to be a place with that name somewhere), but it doesn’t know you’re 200 feet away from the next nearest person who happens to be in the washroom and you’ll never come into contact with.

For that level of granularity, the most widely available technology that we have now is Bluetooth. Low-Energy Bluetooth (BLE) to be exact.

How does mobile phone contact tracing work?

This is the question that kept me awake at night. I want to be a good citizen and participate in this program. I want to know if I have come into contact with an infected person as early as possible so I can isolate myself and protect others. I am also a privacy pundit and the thought of allowing anyone to track my location willingly as is anathema to me, so I was at an impasse. An impasse that would ultimately end up in not participating in the program.

I watch the TWiT (This Week in Tech) podcast which is a weekly foray into technology at a level that people like me want. It’s not fluff produced for easy digestion by the uninformed masses, but it also is not complete gearhead stuff that nobody can understand that’s what Security Now! is for). It is somewhere in the middle and a good source of information for technical people. As you’d expect from such a show, contact tracing has been a hot topic recently.

When I first heard of mobile phone contact tracing, I envisioned a system where my precise location was sent to some central repository every few minutes along with the locations of everyone else. Then, when someone tested positive for COVID-19, whoever is in charge of that giant pool of location data can dig through it to find other people who have crossed the infected person’s path.

This architecture would be ridiculously unwieldy, utterly privacy invading, exactly what I expect of a government program, and simply would not work for a myriad of reasons, some of which Bruce Schneier listed in a recent post. There is no shortage of opponents to the system and they’re not shy about saying it. Ross Anderson illustrates some comical, but entirely likely scenarios:

Anyone who’s worked on abuse will instantly realise that a voluntary app operated by anonymous actors is wide open to trolling. The performance art people will tie a phone to a dog and let it run around the park; the Russians will use the app to run service-denial attacks and spread panic; and little Johnny will self-report symptoms to get the whole school sent home.

He’s not wrong. That’s all going to happen.

Back to how it works: the current thinking in privacy-centric circles is that the best solution to any privacy problem is to collect less data. Data that does not exist cannot be stolen or abused. That makes sense but the problem is that our applications and devices become more complex over time and therefore need more data to be more useful. The best compromise to that problem is to allow the collection of data but leave that data in the users’ possession instead of carting it off to some giant data lake in the sky.

There is ample precedence that user possessed data works. LastPass and Proton Mail are a few examples where user data is only accessible on the user’s device. Yes, the data does exist on internet servers, but that copy of the data is encrypted and cannot be decrypted on those servers. Only the users’ device can decrypt it which means that the useable data only resides on the user’s device.

Here’s a somewhat hard to follow infographic from Apple and Google showing the process.

There are many different organizations working on developing contact tracing apps, but the biggest is a collaborative effort by Google and Apple. Except for a very small group of fringe users, every smartphone on the planet runs Google’s Android operating system, or Apple’s iOS operating system. These two companies can tap into pretty much every phone in existence so this is the logical place to develop contact tracing using a mobile phone.

Google has a terrible reputation for privacy. I won’t hijack this post by pointing out yet again that Google makes almost all its money from advertising; a product line it fuels by surgically harvesting our user data through Gmail, Android, and hundreds of other apps we’ve never heard of. Apple has enjoyed a reputation for privacy recently and, to be fair, it is one of the few companies left that does business the old-fashioned way of just selling us stuff we like instead of selling us data harvesters. But, it is suffering from the same technical debt that occurs every time Steve Jobs leaves the building and there is a steadily increasing number of flaws in iOS and Apple hardware these days.

But, the joint architecture declaration of how these two companies intend to develop a contact tracing app checks all the right privacy boxes.

How will our privacy be protected?

I will now go through the high points of the Apple/Google solution. Please keep in mind that many governments are commissioning their own contact tracing apps. In fact, at the time I am writing this, there are at least 30 countries ramping up contact tracing in different ways. Many of them will not use the Apple/Google solution and their apps may be more invasive. Also, when the Apple/Google solution finally becomes available it may differ from what I’ve written here. But as of today, this is the intention of Apple and Google.

Phase One

Phase one of the project is to develop an Application Programming Interface (API) that can be used for contact tracing and make it available to app developers. That will enable developers to write apps and put them in both the Apple and Google stores. Users can then install these apps and activate them to participate in the contact tracing project.

The reason for phase one is to speed development. The API will be used in phase two as well, but it takes much more time to implement phase two, and a simplified phase one allows us to start building the skateboard.

Phase Two

Phase two of the project is to do away with the apps and build the contact tracing functionality directly into the Android or iOS operating system. In both phases, the user will have to deliberately consent to participate in the program which means even in phase two when you have no choice but to accept the software into your phone, you do not need to enable it.

If you’re participating, regardless of which phase, your phone will generate a unique token every few minutes and use the Bluetooth transceiver in your phone to broadcast that unique token out. At the same time, your phone is listening for other people’s tokens. Everyone else’s phone is doing the same thing so essentially we now have a way of recording the tokens of the devices that are in a ~100m bubble around us.

One of the early criticisms of the Bluetooth solution is that it could not determine how far away a person is, and would not know about things like walls and windows separating people. That is true, but the proposed solution addresses that in two ways:

1. When accepting a token from some other device, your device will consider the strength of the signal from that other device. Weaker signals mean that the other device is farther away or behind something.

2. Tokens from other devices need to be seen by your device more than once in a period of time that will be set by Public Health. For example, that may be set to 5 minutes. That means your device will not record every token it sees, just those that it sees more than once in a 5 minute period. This eliminates a ton of “drive-by” noise.

Those are not perfect solutions to the proximity problem, but they’re the best we have with the technology that most people are carrying in their pockets right now.

Phones will only record the tokens it receives and no other identifying data. Periodically (we assume at least daily) Public Health will download a list of tokens belonging to known infected people to your phone. If any token in that list matches the tokens stored on your device, then you will be notified via the phone that you came into contact with an infected person. What happens next is determined by your local Public Health, but we can assume it will take the form of notifying public health and going in for testing.

From the Apple/Google contact tracing FAQ:

If a match is detected the user will be notified, and if the user has not already downloaded an official public health authority app they will be prompted to download an official app and advised on next steps. Only public health authorities will have access to this technology and their apps must meet specific criteria around privacy, security, and data control.

The two important pieces that ensure user privacy are:

1. You do not upload your token to Public Health. It remains on your device within your control.

2. Only you will be notified if you’ve come into contact with an infected person. Public Health cannot know this on its own because the comparison between the list of tokens you’ve come in contact with and the list of tokens belonging to known infected people happens on your phone, not in “the cloud” somewhere.

There are a few other important concepts in the proposed architecture that I’d like to call out:

1. The contact tracing function on the phone has a very clear edge. It will alert you if you have been in contact with someone who has tested positive for COVID-19 and then it stops. It does not tell you what to do. It does not tell anyone else. This is the “line in the sand” and it is up to the individual and regional health authorities what to do next.

2. The same FAQ states that Apple and Google “will disable the exposure notification system on a regional basis when it is no longer needed.” This is what we call a “forward-looking statement” which is polite speak for “that may not happen.” But, both Apple and Google have proven to be formidable foes in legal tussles and they both have more money than almost any country in the world, so I feel there’s a reasonable chance this will be accurate.

Final Thoughts

I have decided that if the final Apple/Google solution is similar to the current proposal, then I will adopt it when it becomes available. While the criticisms of the false positive rates have some merit, I think the solution mitigates them as much as technically possible. I am also pretty sure that as people start using that data, false-positive rates will inform Apple and Google how to tweak the detection better.

My biggest concern was always the privacy aspect. And now that I know my location data isn’t streaming off my phone to the government or some private development company, I feel much better about that part of it.

my shorter content on the fediverse: https://cosocial.ca/@jonw