The Internet Needs A Red Pill

I consider the internet to be one of humankind’s greatest inventions. Computer networking began as a solution to the perceived problem that a single missile from the Soviet Union could take out the entire United States phone system and render the US unable to coordinate a response. Nobody thought about that before the Soviet Union launched Sputnik and suddenly the entire world was aware that those damn Ruskies were floating around above us doing god-knows-what devious deeds.

In the early ’60s, two things happened in rapid succession. I’ve read almost everything there is read about the origins of the internet and I’m still not sure what the true story is – feel free to search for a version you like. Regardless of the details, around this time, researchers proposed a “galactic network” of computers that could talk to each other. A few years later, another floated the idea of a “packet-switched” network. Packet switching was a necessary technology because it breaks messages into small packets before sending them across the network and the receiving system re-combines them into the original message. The critical part of packet switching is that it allows each little packet to travel to its destination by different routes if needed, thus ensuring that a single strike taking down a portion of the “galactic network” would not prevent packets from arriving at their destination via another route.

Although the capacity, size, and underlying technologies have improved over the years, the same basic premise has not: interconnected computers sending discreet small packets over the best routes. It is the “best routes” thing that I will be talking about in this article.

A primer on BGP

Despite popular phrases like “log in” and “surf”, we do not “go onto” the internet. Rather, we use the internet to transfer data into our computers. When you log into a website, you don’t “go to” that web site’s login page; you pull a copy of that login page into your computer, fill in the details, and then you send the whole thing back to the remote web server. Every bit of that page is loaded from some computer connected to the internet somewhere, and when you click the login button, all that stuff gets sent back to some other remote computer connected to the internet. But, how does that happen? How does your computer’s request for that login page find the correct destination computer in the mass of trillions of computers all connected to the internet? The short answer is that the remote computer has a unique IP address, as does your computer, so they can both be uniquely identified despite the overwhelming number of computers comprising the internet.

But that is not the whole answer. Just because my computer knows that gmail.com has IP address 172.217.0.5 doesn’t mean my computer knows where that computer is physically connected to the internet. My computer has absolutely no clue where that IP address, in fact. It just issues a request for the GMail login page to that IP address and then sits back to see what happens.

The internet takes care of figuring out where 172.217.0.5 is plugged in so we do not have to worry about it. The movie The Matrix had a nice solution to this: the red pill. The blue pill would put Neo back to sleep in the virtual world the machines created and he would remain blissfully ignorant to the existence of the Matrix. The red pill, however, would pull him out of the Matrix by acting as a locator beacon for Neo’s physical body in the fields, allowing Morpheus’ crew to find him and physically unplug him. Sadly, we can’t ask GMail to take a red pill for us, so we have to rely on the intelligent routing capabilities of the internet; primarily, the Border Gateway Protocol (BPG).

The internet is comprised of millions of little systems named Autonomous Systems (AS), which is just a paper record of who owns and controls what IP ranges. Any organization that wishes to own IP addresses must first obtain their own AS from one of the Regional Internet Registries and then obtain IP addresses to associate with their AS. All major internet players all have their own AS and they tell the rest of the internet what IP address they control by announcing those IPs using BGP. So, when you type gmail.com into your browser bar, the following process occurs:

1. Your computer asks the Domain Name System (DNS) for the IP address of gmail.com which is 172.217.0.5.

2. Your computer then splits your request into many different packets and sends them out that blue wire, or your wifi connection, onto the internet, with 172.217.0.5 stated as the destination.

3. Your packets go through many internet devices like switches and routers on the way to its destination. It is those devices that have knowledge of where IP addresses are physically plugged into the internet, much like Neo. They speak the Border Gateway Protocol to other devices and share information such as “I know where 172.217.0.5 is and it will take 5 hops to get there”. Another switch may say “I know where 172.217.0.5 and it will take 3 hops to get there”. In which case, most routing devices will choose to send the packet along the shorter route of fewer hops so it gets there faster.

4. The internet is huge and ever-changing. No device can hold the entire IP allocation table of the whole internet. Therefore, a lot of devices do not know where 172.217.0.5 is located. In those cases, the packet is forwarded “up” through larger and larger ASs until some device says “I know where 172.217.0.5 is and ….”

5. This process happens for every packet. Your login username and password can amount to hundreds of individual packets and each one of those could, potentially, take a different route to GMail. In practice, these routing devices cache BPG results for short periods, so subsequent packets should arrive quicker than the initial one and usually by the same route unless the routing information has changed, such as sudden damage along the way.

6. Your computer is blissfully unaware of all this. Once it is done sending packets, it forgets about the whole affair and goes back to doing whatever computers do when they’re idle.

7. Eventually, reply packets start coming back to your computer from the GMail server following the same routing process as before. Your computer reassembles the packets and eventually, you are rewarded by the GMail inbox loading in your browser. Hooray.

8. Now you click an email to read it and the whole thing starts all over again.

BPG hijacking

BGP is an absolutely critical part of how the internet functions and when it breaks, it can create very large outages that sometimes can be very hard to fix. Strangely, despite its importance to the basic functioning of the internet, BGP has no security built into it. Remember those ASes that major internet players have? They can announce any IP address range they want and once those announcements are copied out to the routers on the internet, then some of the traffic destined for those IPs will be routed to that AS’s servers even if that AS does not legitimately own the IPs it is announcing. This is called BGP Hijacking and it’s a real thing.

Stay with me here: BPG Hijacking means I can configure my switch to announce Gmail’s IP range 172.217.0.5 and then your GMail requests will start coming to me instead of Google.

This has huge security implications. If I can route all your banking, government, and medical information to my servers then not only am I bringing those legitimate services offline, but I may also find myself the recipient of lots of private, juicy information that I may be able to use for evil. This may sound like rare circumstance, but BGP Hijacking is more common than we’d like. Sometimes it happens by accident, but other times a nation-state actor like China routes all of Google’s traffic into China which doesn’t sound like an accident to anyone.

Preventing BPG hijacks

Recall that AS stands for Autonomous System. The keyword here being Autonomous. Nobody can tell me what IP ranges to announce from my AS. I can be as nefarious and incompetent as I like, and the internet will feel it as traffic goes haywire. The best protection we have for this at the moment is called Resource Private Key Infrastructure (RPKI). This is a system whereby I, as the legitimate owner of an IP address space, can sign my IP announcements with my private encryption key which proves that I am the legitimate owner announcing that IP range. If anyone else decides to announce my IP range, those announcements will not be signed because that person (hopefully) does not have my private key. That is a good start, but signing my announcements does nothing by itself.

Although we consider the internet to be a virtual world, it actually is very physical. Internet traffic travels through physical cables and ends up at physical servers. The companies that own those cables are called transit providers and they are a critical component of BPG security. In order for RPKI to work, every transit provider has to do two things:

1. implement checks on BGP announcements to see if the AS announcing the range has signed those announcements, and

2. confirm that announcements for that IP range should be signed.

This is where the BPG Hijack prevention mechanisms start to break down.

Not all AS owners set up RPKI which means transit providers cannot assume that every IP range announcement should be signed. Further, implementing RPKI costs transit providers something. They have to deploy extra software to manage RPKI and, honestly, there’s very little incentive for them to do so. Transit providers are content-neutral; to use a metaphor: they are just the highway – they have nothing to do with the cars driving on it. Consequently, many transit providers have not implemented RPKI which means it is not very effective, even if an AS owner has done their part and set up signed announcements (called ROA -Routing Origin Authorization records) for its IP ranges.

The good news is that we’re getting there. Like in any industry, there are big and small players. Most of the big transit providers which comprise a good chunk of the main internet routes, called the backbone, support RPKI. This has the effect of lessening the impact of BPG hijack attempts because those illegitimate announcements can’t spread too far before they hit a major transit provider supporting RPKI that will drop that announcement. However, regional problems are still possible.

But, we are still not there yet. If you want to get a sense of how fragmented the RPKI effort is, take a look at this site, Is BGP Safe Yet? Spoiler: no.

Scroll down a bit to the Status section and click the Show All link. You can quickly see that the major backbone providers (most of which you have probably never heard of) are marked safe because they are doing both things: a) checking for signed ROAs and b) dropping announcements that are not signed properly. But as you go down the list you will see a lot of “partial” listings which are transit provider only doing one of those things, and finally, you will see a lot of smaller players that you may start to recognize as consumer internet service providers, which are not implementing RPKI at all. That list is by no means exhaustive, but it gives you a good sense of how far away we are from safe internet.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

When You Don't Need Security Is When You Need Security

I worked at a number of jobs when I was younger before I finally ended up in my technology career. I think most people go through a bunch of similar jobs growing up such as working in the hospitality industry. While I did have a burgeoning career in cooking, I left that at one point and got into physical security. I joined the army reserves (AKA “the militia”, but not to be confused with the American-style prepper gun nut militias) as a part-time job, and for my full-time job, I worked as a security guard. I’m not in the physical security field anymore, but there’s a lesson I learned during that time that has stuck with me for my entire career and still serves me well today in the information security field and it’s the topic of this post.

When I was toiling away in the middle of the night as a security guard at hospitals and truck companies, my supervisor would stop in a few times a night. He’d always want to see my rounds report and he would always tell me to put more detail into it. I’d say “but there’s nothing to say, nothing happened”. He’d reply “Nothing happened because there’s a security guard roaming around randomly looking for trouble. We need to make sure the client knows that.”

Eventually, I got out of my security guard mindset and started thinking like a manager. I don’t know who hired our company to provide security at these companies, but someone did. And that person has to justify that cost to someone periodically. I started to realize that the measure of success for that person’s decision is that NOTHING HAPPENS. But you know what’s really hard to justify at budget meetings? Spending money on something when NOTHING HAPPENS. I started putting more detail into my reports at that point because I started envisioning that person defending my wage. I started recording when I noticed a padlock unlocked, or a car I hadn’t seen before driving by the compound, or an office door open that usually is not. I wanted to give that person a good idea of what they are paying for and give them the tools to continue paying me. I envisioned the finance people at the meeting denying the security budget request because “we don’t need that, nothing happens anyhow.”

I envisioned the finance people at the meeting denying the security budget request because “we don’t need that, nothing happens anyhow.”

Infectious Disease

This problem isn’t just a security problem, it exists in many areas. Anti-vaxxers are an example. In recent memory, anti-vaxxers decided that inoculating the population against measles was no longer necessary because nobody has measles anymore. Well, guess what happens when you remove the thing that is making NOTHING HAPPEN? You get a measles outbreak in one of the most medically advanced countries in the world. (CDC)

The majority of cases were among people who were not vaccinated against measles

Nothing was happening because something very important was happening – vaccinations.

Infosec

It’s hard to put a value on security. That is true in the physical security world and it is true in the infosec world. When security works, nothing happens, and it’s hard to predict what could have happened if the security was not in place and the value of the damage that this thing-that-did-not-happen could have caused.

Infosec prevents things from happening by employing the “kill chain”. Yes, the “Cyber Kill-Chain” (or Intrusion Kill Chain) is a model, just like its roots in the military kill chain. It has problems, but so does the OSI and we use that all the time. Infosec is effective because it severs the kill chain at the earliest possible opportunity which renders the attacker unable to complete its mission. (Sucuri)

Infosec “security guards” identify reconnaissance activity and mitigate it, therefore NOTHING HAPPENS because the attacker cannot get enough intel to proceed to step two and weaponize an attack effectively. At my work, we sometimes see customers decide to let their Web Application Firewall (WAF) subscription lapse; some of them actually cite cost as the factor. A measurable chunk of them come back a month later with an infected website. The WAF broke the kill chain for them, and when they removed it, the attackers were able to advance their attack successfully. But because the WAF as working, the perception is that NOTHING HAPPENED so there was no monetary value in maintaining the subscription.

Reporting

These are but a few examples to show that reporting matters. A lack of incidents doesn’t get noticed; we need to find a way to report the unnoticeable things that are preventing noticeable things from happening. The CDC graph and narrative does a good job of this: it shows that there is a correlation between measle outbreaks and a drop in vaccinations. It is much harder to do this with infosec because it’s more difficult to determine what is a deliberate recon and what is just weird traffic. And, even more importantly, if you are reporting an incident further along the kill chain than a reconnaissance, you’re effectively reporting that something DID happen despite the measures in place, but was still mitigated at later steps.

It’s no secret that I use the Sucuri WAF to protect my sites and we go out of our way to provide reporting to our customers so they know there’s a reason that NOTHING IS HAPPENING to their site.

Sum up!

Infosec workers and managers need to provide some level of reporting to stakeholders. It can be tough to do because many times when something trips an IDS, infosec workers investigate and determine that it was just a harmless port scan and close the alert. But that port scan should be recorded somewhere; not because it may escalate later (although it may), but because stakeholders need to know that they were port scanned, they were targetted, and the defence measures in place mitigated the attack right at the first link of the kill chain. That is what stakeholders are paying for so they need to know they’re getting something for their money.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Adobe Flash killed the railroad star...

I’m always mystified by people who resign from committees or other roles because they don’t like how things are going. Resigning because the vote didn’t go their way, or because they think the organization will be pulled into disrepute because of its actions does not make me think highly of that person. It’s much harder to make change from without. People who really want to make change don’t give up their seat at the table because that only serves to relinquish all their power to make those changes. Resignation is for chumps.

• * *

How many cows?

No. Bad, Jon. Not a good joke.

Back in the day, TUCOWS was the place to download “shareware”, the original digital licensing system where you could “try before you buy”. Software back then was rarely completely free. The Open Source Movement hadn’t really caught on and the internet wasn’t developed enough to distribute software anyhow. Shareware was software that you were encouraged to share because it had limitations. If you decided to pay the licensing fee, everything would be unlocked for you, but the unlocked version could be shared with wild abandon. We call that the “Freemium” model these days, but it’s been a thing for a long time.

Back to my bad joke. TUCOWS was an acronym for The Ultimate Collection Of Windows Software. It predates Linux, and Macs were so rare back then that I had only ever seen one in real life at that point. It belonged to another geeky friend of mine that I met on Nucleus BBS in Calgary, and he ran his own BBS on that Mac using Hermes BBS software. Yes. I am that old.

Sadly, TUCOWS is shutting down. Honestly, it probably hasn’t had a visit in years, so we should be giving Tucows (note the capitalization change) props for keeping it running for so long despite being a money pit. Thanks for all the downloads, cow folk!

Unintended Consequences

Regular readers will know that I love unintended consequences. Our world is interlocked very tightly with far-flung places and people. We’ve learned a bit of that during the COVID-19 pandemic; people home sick in other countries manifests as shortages of supplies in our country. We really are all in this together.

This little story has a nice tech angle to it. Adobe Flash has been an internet security nightmare for years. It is easily the least secure part of our computers and its architecture is such that it is basically impossible to fix. It is so bad that I even wrote an article specifically about how bad Flash is back in 2017.

Well, Adobe finally did the entire world a solid and killed it off. Flash is no longer available and if you still have it installed, it has stopped working (you hope). But this section isn’t about security, it’s about unintended consequences. Like the fact that the railroad in Dalian, China, relied entirely on Flash for its operations and the whole railroad ground to a halt when Flash hit its end of life.

Guess how the railroad techs fixed it. Go on, guess….

They installed a pirated version of Flash!

Yep. I guess they went to the “when someone removes a dangerous weapon pointing at you, it’s best to go find an even more dangerous weapon and point it right back at yourself” school of technology. I can’t even.

Please remove Chrome. I’m begging you

Google’s Chrome browser has been popular for a number of years. It is fast, it is sexy. It is not Firefox. And it is really just another tool Google uses to harvest your private user data under your nose.

Regardless of what you may think Google’s business is, unless you said “advertising”, you’d be wrong. Google derives the majority of its income from selling ads to you while you search and read Gmail, and it is able to sell those ads because it has just oodles of personal data on you, so it’s knows what you want.

How bad is Google? I recently stopped using Google Pay because I found out that Google would tell advertisers when one of their ads was shown to me, and then saw that I bought something related IN PERSON AT THE MALL. What the hell, Google?

I get that reporting that I clicked on an ad and bought something online is an obvious no-brainer that I should expect. But I should not expect Google to take my real life, in person, walking in the mall without even looking at my phone, credit card purchases and give them to some random advertiser. That, dear reader, is utter dystopian bullshit.

Google has taken yet another step down into the irredeemable pit of its own making by preventing users from installing ad blockers in an upcoming version of Google Chrome. Because they sell ads. Get it?

Before you take the seemingly reasonable stance that viewing ads is how you “pay” for content, let me assure you that ads are one of the most highly dangerous malware delivery mechanisms known to man. Seriously…this is such a problem there’s even a term for it: Malvertising. There’s a pretty good chance that the ad you’re looking at is actually malware trying to infect your computer, although much less now that Flash is dead. A much better way to block ads is to do it at the DNS level using a tool like Clean Browsing (disclosure: run by friends, but I am not involved).

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Unlike Windows or macOS, Linux is a community-based operating system. It has a lot of flavors, called “distributions” which are customized in different ways, but at its core, most Linuxes use basically the same applications, called “packages” to get stuff done. Because there’s no corporate bean counter overseeing the efficiency of the whole operation, the Linux community has been able to retain its culture beginning from the latter half of the 20th century to today. Some of that culture is contained in the inexplicably weird decisions made about packages. Cowsay is one such package.

Cowsay is a program that prints a little ASCII cow on your screen with a speech bubble stating whatever you want. It has absolutely no value other than that; it’s just one of the whimsical packages that are still included in nearly every Linux distribution in existence.

ASCII, pronounced “askee”, art has a long proud tradition on the internet and is a short form for the American Standard Code for Information Interchange. Today, we use the Unicode character set which allows graphical stuff like emojis to be displayed on our screens, and ASCII is now folded into the larger Unicode spec. But back in the days before Unicode, we had to make our own art with the extremely limiting 128 characters on our keyboard.

ASCII art of the Apple logo (https://cmp.cx/ae899)

The primary use of ASCII art was to adorn dial-up bulletin boards (BBSes) with pretty login screens and menus. Later, those screens and menus became much prettier with the advent of ANSI, (American National Standards Institute ) pronounced “ansee” art. The ANSI spec has twice as many characters as ASCII, but most of them are not visible to us. Rather, they are used to control the visible text in ways we can see, such as changing its color.

ANSI art example from the Hook-Ups BBS (https://cmp.cx/02e69)

OK, that’s enough background, let’s get back to cowsay. If you have a Linux box, type cowsay Hi! and see what happens. If nothing happens, use your package manager to install cowsay and try again.

Now let’s have some fun. Try cowsay -l (that is lower-case L) to get a list of things other than cows that can say what you want. Use the -f switch to specify a different, non-cowlike, messenger.

└──╼ $cowsay -l Cow files in /usr/share/cowsay/cows: apt bud-frogs bunny calvin cheese cock cower daemon default dragon dragon-and-cow duck elephant elephant-in-snake eyes flaming-sheep fox ghostbusters gnu hellokitty kangaroo kiss koala kosh luke-koala mech-and-cow milk moofasa moose pony pony-smaller ren sheep skeleton snowman stegosaurus stimpy suse three-eyes turkey turtle tux unipony unipony-smaller vader vader-koala www

Even if you like the cow, you can change its appearance with some simple switches:

cowsay [-e eyestring] [-f cowfile] [-h] [-l] [-n] [-T tonguestring] [-W column] [-bdgpstwy]

It’s those last options that bring about the most visible changes, but you can change the cow’s eyes and give it a tongue with the built-in switches as well. If you don’t think cows should speak, you can use cowthink instead and the speech bubble will be replaced by a thinking bubble.

└──╼ $cowsay Hi! _____ < Hi! > ——- \ ^^ \ (oo)_____ (__)\ )\/\ ||——w | || ||

└──╼ $cowthink Hi! _____ ( Hi! ) ——- o ^^ o (oo)_____ (__)\ )\/\ ||——w | || ||

Tony Monroe wrote cowsay over two decades ago and it is still a mainstay of the nerd herd. There are versions of cowsay for every operating system, libraries to include cowsay into your own application, and even web-based cows to help those of you who haven’t yet found your way to the superior *nix-based operating systems. Microsoft even sells a version of cowsay. Talk about a proprietary systems tax!

Go forth and cow, now.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Fiber Data Connections: LC vs SC

If you’re working in data centers, you’re going to need to know something about fiber optic connections. Fiber is the most common type of connection used for that last mile between backbone providers and your equipment. Just like any other cable, fiber cables have to connect to your equipment and this article is about two commonly used fiber connectors: the SC and LC connectors.

A quick primer

First, some background. My exposure to data centers is generally a single cabinet in general population. My DC provider gives me rack, power, and a patch panel. The patch panel is the central part of this article.

The patch panel is generally mounted at the top of the rack because most data centers run their cables over head. The part of the patch panel that I use is referred to as the A-side and the other side is referred to as the Z-side.

When I deploy into a new DC, my backbone provider gives me a Letter of Authorization (LOA). I take that LOA and order a cross connect from the DC using that letter. The DC techs need that LOA because it is their authority to connect my backbone provider to my patch panel. The DC techs will run cable from the ports my backbone provider has specified on their patch panel in the bowels of the DC (the Z side) to the ports I dictate on the patch panel in my rack (the A side). I then show up and start plugging things into my patch panel (the A side).

Connectors matter (kinda)

The LOA will specify whether the DC techs should use LC or SC connectors on the Z side and I can usually instruct the DC what type of connectors I want on my end. It is important for me to know what type of connectors are being installed on my patch panel because I need to bring a compatible set of pig tails with me. More on that later.

There are really only two considerations when choosing between LC and SC connectors, in my opinion.

• SC connectors are bigger and will take up an entire port on my patch panel.

• LC connectors are exactly half the size of an SC connector and I can therefore fit twice as many connections on my patch panel.

If you are operating in a high density patch panel configuration, you may want to chose LC solely for the reason that you can plug more stuff in.

A third consideration which doesn’t have a lot of bearing for me is the mating cycles of each type of connector. A mating cycle is the entirely too apropos name to describe removing and re-inserting the connector. Fibre is glass and the connectors are porcelain. Both of those materials aren’t terribly resistant to abuse and each push/pull wears the porcelain ferules down just a little bit more until they finally will not be able to transmit the light well. An LC connector is rated for 500 mating cycles in its lifetime whereas an SC connector is rated for 1,000 mating cycles.

Initially LC connectors were much more expensive due to licensing fees. Now the costs are not a consideration unless you’re buying millions of them.

In my situation, I install a data center once and may never go back to it, so I’ll never come anywhere near 500 mating cycles. Consequently, that is not a concern for me. If you’re in a situation where you’re constantly shuffling fiber cables around, then you may care more.

Practical information

Here are pictures of each type of connector plugged into a patch panel.

LC connectors

The image above shows LC connectors and you can see that each port on the patch panel can take two LC connectors.

SC connectors

This image (above) shows SC connectors and illustrates how each SC connector requires its own port.

Finally, on to the pig tails pictured below. The most common use for fiber connections is to run the fiber channel from the patch panel into a switch. Many switches use MPO (Multi fiber Push On) connectors. There is also a newer type of connector called MTP which is the same form factor, but more advanced.

Pigtail with MPO

The loose blue ends are LC connectors and the green end is the MPO connector. the MPO goes into the adapter pictured on the right, and then into an optical port on my switch. The LC ends go into the patch panel which is why it is important for me to know what type of connectors the DC techs installed into my patch panel.

And that’s the 25-cent tour of fiber connections in a data center.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Building Stuff Without Building Stuff

I’ve always been a tinkerer, a hacker, and an inventor of cobbled-together things that “Just Work”. Throughout my life, I have learned to code, build databases, provision servers, root mobile devices, and whatever else was required to get things done. These days, technology has advanced to the point where it’s possible to make complex and beautiful things without getting your hands dirty at all.

There are two similar terms running around the internet recently: low code and no code. These terms refer to the ability to build applications or services without writing much, or any code. This article is not about the low/no-code movement. While that movement is super interesting in its own right, I am going to talk about tools a little higher up the stack where I build processes and tools but stop short of building an app.

I am always trying out new services to see how they work and if they should have a place in my busy life. Most tools I discard because while the intentions of the developers may be good, there are just too many rough edges on most applications I demo. Two notable exceptions to that are Zapier and Airtable. I find both those services to be exceptional at what they do. So much that I’m dedicating an entire article to some simple, but very useful integrations I built to make my life easier. Let’s start with Airtable.

Databases

I don’t think many people have direct experience with databases. Although probably close to 100% of applications and services use databases internally, I think end-users generally don’t use databases directly at all, or they think they do when they open up a spreadsheet. Spreadsheets are not databases, although both can do calculations on data. The real power of a database is the ability to breakdown, retrieve, and analyze the data in it. Spreadsheets have developed hackey add on functionality over the years to make them more database-like, but most people who cross that line and try a proper database have a big “wow” moment and never go back to the spreadsheet.

Airtable DBaaS (Database as a Service)

I made up the DBaaS bit, but that’s what Airtable essentially is. It is a powerful, but very polished cloud-based database service. Airtable fools you at first because the default view for a new database table is a grid view which resembles the ol’ Excel spreadsheet. But, it’s not and does not work like a spreadsheet.

The first thing I did with Airtable was a replacement project. I had grown tired of all the “todo” apps on the market, but I definitely need one. I subscribe to the Getting Things Done (GTD) framework to keep my life organized and although many todo apps support GTD, they bolt other stuff onto it that makes it unwieldy or, even worse, cause distractions for me. A good example of unneeded and distracting functionality are time notifications. If something needs to happen at a specific time, then it is not a todo item; it is a calendar item, therefore alerts have no business being in a todo app.

Because I wasn’t able to find anything that did exactly what I wanted, I built one in Airtable.

This image isn’t actually my todo database. There’s far too much personal information in mine to make a decent screenshot from, but this shot shows many of the features that I use. I use dates and priorities to help sort what is important on any given day. I also heavily use grouping and sorting to create useful views. I have a work view and a personal view, and other views for groups I belong to. I also have views for completed items so I can go back and see when a task was completed rather than simply deleting it outright.

The Airtable Universe

Where did I get this example todo list that shows all these features, you ask? From one of the most amazing things about Airtable: the Airtable Universe.

The Airtable Universe is a community repository of databases that people and companies have created as a springboard to help Airtable users get started. There are hundreds, perhaps thousands of databases in the Universe that can be copied into your Airtable account with a single click. To be honest, I have not used any of the Universe databases yet, but I am sure I will at some point. So far, my use of Airtable has been to migrate many existing apps and documents. The structure of these databases was already set making it easier to start from scratch.

Mobility

I don’t particularly need all my databases in my pocket, but certainly, it is essential to be able to access my todo list at any moment. The Airtable mobile app allows that.

The app only supports a subset of Airtable features. The most notable feature missing is the grouping feature but I understand why. Grouping is a visual tool to help us see meaning more clearly; “these people have paid their dues, these people have not” type of thing. That type of analysis necessarily takes up a lot of screen space; space that is not easily available on mobile devices so it’s been omitted entirely. However, viewing and updating records works perfectly which is most of what I need on the road. The fancy views and printed reports can wait until I am on my laptop.

Getting data into Airtable

I’ve been using databases for a long time. One of the recurring pain points with moving an existing set of data to a database is the data entry. Databases shine at analyzing data, but you have to get that data in there in the first place. Airtable allows the import of a CSV which is a useful option. Let’s face it, a lot of us have spreadsheets kicking around that have grown into these Frankenstein monsters stumbling around our hard drives. Import that sucker into Airtable and you’ll instantly be able to sort and discover that data in new and novel ways.

Another great way to get data into Airtable is to let something else do it. For my day to day entries, I let Zapier do some of my data entry.

A note on Airtable security

I work in infosec and the first question I always ask of any service pertains to security. Almost everyone uses TLS (HTTPS) these days, but that’s not sufficient in itself. HTTPS secures data during transit which is important, but it is not the whole enchilada. So, I investigated and I was happy with what I found about Airtable’s security model.

Airtable uses HTTPS (as mentioned). It also supports non-SMS 2FA to help secure your login and – I was surprised and happy to see this one – Airtable encrypts my data at rest and also within its backups.

Nice.

Zapier AaaS (Automation as a Service)

I sincerely hope AaaS doesn’t become an actual term because..well…how would you pronounce it, exactly?

Zapier’s reason for living is to connect disparate apps together so they can exchange data. This integration is done through the use of “Zaps” in Zapier parlance. Want to record every Twitter mention in a Google sheet? Want to send an SMS message when someone updates a Trello card? Want to…whatever, you get my point. The folks at Zapier spend all their time integrating services so they can talk to each other.

I use Zapier in two small, but useful ways.

Google calendar –> Airtable todo base

Despite my assertion that tasks with times on them are not “todo” items, I do want to see my calendar items for the day in my todo list, just not with times and alerts going off.

I created a calendar in my Google account named Airtable. When I put a new item into that calendar, Zapier notices and it sends that item to my todo list in Airtable. But that is not all there is to it. It would be very easy for me to forget that new todo item because it is not yet filed into a category that I sort or group on. So there’s a step two…

When that new item is added to my todo list, it goes into an Airtable view I have called “New” which is where items that have not been categorized yet go. Think of it as the “Stuff” list from the GTD methodology. I manually look through all my todos every day (again, as part of the GTD methodology) so I will eventually see/remember it and categorize it, but I want more. So, I have a second Zap that runs on that view in Airtable.

When a new item is added to that view, Zapier notices and sends me an SMS message reminding me that there is a new todo item to be categorized. This means I have two Zaps running just to get stuff from Google calendar into Airtable, but who cares. It is all automated and I never think about it anyhow. Also worth noting is that I am a free Zapier user, but if I subscribed, I would gain access to multi-step Zaps and then these two Zaps could be a single multi-step Zap.

Email tag –> Airtable correspondence log base

I am the secretary for a community organization and one of my many duties is to maintain a log of all correspondence that comes in or goes out of our group. I have an Airtable base for this, but again – it’s all about getting the data into it. Because the vast majority of our correspondence comes in via email, this screams to be Zapped, so I did.

Interesting to note is that I can’t use Gmail for this. While Zapier does have some GMail integrations, it can’t do this particular one. No problem, I moved my secretary account to Zoho where Zapier can work its magic.

Nothing initially happens when an email arrives. I did this on purpose because I don’t necessarily want every email that comes into that inbox to be part of the correspondence log – there are lots of internal emails that don’t qualify for the log. To gain control over which emails end up in the correspondence log, I configured my Zap to only grab emails that have a certain tag on them.

Part of my daily routine is to review that email inbox and if there are any emails that should form part of the correspondence log, I tag them. Zapier notices that and inserts them into my correspondence log in Airtable.

Go forth and automate!

There are probably literally a million things you can do with Zapier. I’ve just scratched the surface with the few things that really do make my life easier. Zapier gives you 5 zaps and 100 executions per month on the free plan which is enough for me. If you need more, it’s time to pay and I would do that if I hit a limit because Zapier is so incredibly useful. I hope this article gives you some ideas to play with and make your life a little easier, too.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Building Bicycles and Drawing F*ing Owls

This week’s post is a bit late because we’ve just come off the “May 24” weekend in Canada and I was working on some house projects that took a little longer than I hoped. These projects were not at all technical – mostly yard work and fence repair, but as I was planning it out and executing it, I realized I was planning it just like I plan projects at work.

• * *

Software and other IT engineering stuff are usually built using the Software Development Life Cycle (SDLC). The SLDC instructs developers and managers to follow a process in order to create good software. They analyze the requirements, design something to meet them, deploy it, and then look after it for the rest of its life. If the product needs to be modified at some point in its maintenance cycle, then the process starts all over again with analysis.

By Dzonatas – Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=4376189

The broad strokes have not changed and likely never will. It’s rather impossible to design something before knowing what its intended purpose is, and you can’t deploy something that has not been designed. Modern software development still follows these broad strokes, but in a much more flexible and granular way.

The original implementations of the SDLC were successful enough, but have changed drastically in recent years; mostly to address a lack of flexibility in earlier iterations. These days we know that any single vision of a product is not going to survive the development process. We know that we’re going to start out building X but wind up building Z by the time the product is ready for the market. The monolithic SDLC doesn’t support this reality very well because it doesn’t have a cycle built into it for modifications prior to deployment (called implementation in the diagram above).

The development industry is filled with stories of spectacular budget or schedule overruns due primarily to this one problem. In mid-stream, a stakeholder will decide that a few features should be added to the project. Because there is no codified way to analyze and design ad-hoc modifications, they usually end up being discarded or bolted onto the project in an inelegant way, typically breaking other parts of the initially well-designed system.

It is the SDLC that we can largely credit with giving us the concept of software “versions”. When a modification comes up, because the monolithic SDLC isn’t great at handling that, it gets marked for “the next version” and forgotten about. Perhaps that next version is years away, or perhaps it never gets released at all. In either case, we needed a better way to incorporate modifications to a product.

These days, most software is developed using an ongoing iteration of the SDLC process which is collectively and generically referred to as “agile technologies”. There are a number of development variations that fall under the agile umbrella. There’s no right way to be agile, although certainly some best practices have been ironed out. In general, agile teams work in a much less formal environment than is proscribed by the SDLC. While all of the steps still occur (analysis, design, etc.) they are done in much smaller iterations, much quicker, and therefore are much more robust to change. New features and previously unknown modifications can be worked into the application much earlier than “next version” and it can be done properly so the product doesn’t have rough edges.

Build smaller, working things

Using agile methods, smaller and less functional versions of the application can be released and built upon. With the initial SDLC, the product doesn’t leave the bullpen until it is a fully complete product that meets all the requirements laid out way back in step one weeks, sometimes months, before.

If software were a car, we’d say that the first thing to be delivered is a fully functional car. That’s great because we set out to build a car and now people can buy it and go grocery shopping and drive to work. What's not great is that those people had no way at all to go shopping or commute while waiting for this car. And, now that they have a car for the first time, they’re asking for all sorts of stuff that was never part of the design.

In agile, we eventually deliver a car, but we also deliver usable bits of the project as we go along. For example, if this car was developed using agile methodologies then we’d deliver a skateboard in the first week. Then a scooter a few weeks later. We’d deliver a bicycle, then a motorcycle and finally, a car. It still takes a long time to get a car, but in the meantime, people can still get to work on the skateboard or scooter. You can’t go grocery shopping yet, but you can do some of the things you need a car for.

Another advantage of continuous development is that it is much easier to incorporate modifications. This is partially because there are multiple and ongoing ‘analysis/design’ phases to accommodate requests for features, but the big win comes from getting the product in front of the end-users quickly. Perhaps the original design did not call for any mirrors on the car but users really want mirrors. Because that was not picked up in the analysis stage, the first time developers are hearing about mirrors is after the launch of Car 1.0 with much fanfare and press. Mirrors will have to wait until version 2.0.

But, consider when the need for mirrors would be discovered using agile. Perhaps the skateboard riders would not come back with an idea for mirrors, but probably by the time users are riding the scooter in traffic they’d start clamoring for mirrors. The agile team can take this feedback, run through the analysis/design phase, and possibly get mirrors on the next step, the motorcycle. By the time the car comes out, we’ve had mirrors for a long time. This well-known image from Henrik Kniberg illustrates the difference between the two methodologies well.

Agile utilizes the concept of continuous delivery of a “Minimum Viable Project (MPV)” instead of focussing on the car. From Kinberg’s post:

Here we take a very different approach. We start with the same context – the customer ordered a car. But this time we don’t just build a car. Instead we focus on the underlying need the customer wants fulfilled. Turns out that his underlying need is “I need to get from A to B faster”, and Car is just one possible solution to that. Remember, car is just a metaphor, think any kind of customized product development situation.

So the team delivers the smallest thing they can think of that will get the customer testing things and giving us feedback. Some might call it an MVP (Minimum Viable Product)…

Notice the subtlety in the image. The end product, the car, looks different. If you read Kniberg’s blog post you will find out that is because, during the bicycle phase, one of the pieces of feedback the users gave was that they like the wind on their faces. So, while the lack of a removable roof may not have been a deal-breaker for the initial model of car, the engineers were able to make a better car because of that early feedback.

Don’t draw the f*ing owl

There’s another saying that I really like and it can be applied to poor software development. I am not saying that any particular development methodology is inherently bad, but there are certainly a lot of examples of software development gone wrong. The main area where software falls down is in the requirements and design phases. If those phrases are not done properly, then there’s a lot of confusion between step one and the finished product. When a roadmap is unclear like that, you’re “drawing the owl.”

To be more clear, the phrase is usually “draw the f*ing owl” because it’s never used in a good way. It’s kind of like the phrase “dumpster fire”. Nobody has ever been happy with a dumpster fire.

The phrase refers to a (hopefully) hypothetical situation where you’re being taught how to draw an owl. But with a ton of missing information, yet you’re still expected to draw the owl. There’s a lot of variations on it, but this is my favorite.

Easy, right? Just lay down some framework code and then just shut up and figure out how to draw the rest of the owl.

Using this method, we won’t find out that we have no clue how to draw an owl until our customer is expecting delivery of the owl drawing. Using a more agile development methodology, we’d find out as soon as we tried to stick the ears on the circles that we are lacking expertise in owl drawing. We could then readjust our course to either revisit the design to see if we really need an owl at all or hire a primo owl drawer to help us out.

What does this have to do with fences?

Maybe not a lot. But all multi-step complex tasks can benefit from simplicity. And a very reliable way to create simplicity is to be agile and develop small bits of deliverable functionality at each step.

Removing the old fence is a good start. It’s a single step with an identifiable and usable delivery: cleared land for the new fence.

Hammering in some small 2x2 stakes along a plumb line is a good second step. It shows us that the new fenceline will be straight, and it also tells us things about the new fence, such as how many fence posts we’ll need, and how high they need to be to accommodate the slope of the ground.

Digging the holes and putting in the fence posts gets us to the point where we can confirm what we learned in step 2. We can also string a rope between the fence posts now to give us a partial fence. We have our bicycle.

Putting the cross beams and fence boards up is a major delivery because we have a working fence. There’s no gate yet, but we have most of what we need, we have our motorcycle.

Finally, the gate seals the deal and the fence is complete. Voila – the car.

While not every step in fence building has a “skateboard” deliverable attached to it, it provides ample opportunities to review frequently and adjust for the next steps if needed. At the end of each step, I learned something – I have too many fence boards and not the right type of hinges are examples of two things I learned during some of these phases. But I learned them early so by the time I needed the correct hinges, I had them so the gate went on flawlessly on time and on budget.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Subscription blindness: Not a thing.

I fixed a lot of stuff today. My furnace broke and while I am not crazy enough to try to fix the fireball-maker in my basement, I did fix the (also) broken thermostat. On the heels of that tremendous success, I then fixed a wall clock and even a Soda Stream. None of that stuff is terribly technical, but the process I used to fix ‘em all was. Problems with things you know nothing about are hard to fix. But in my line of work, that’s all we do – solve big hairy problems that have no visible solution. The trick is to break the thing down into a bunch of smaller parts, and then work your way through those smaller, easier, bits until you stumble across the one that’s broken. That works for software, and it works for thermostats, wall clocks, and Soda Streams.

Subscription blindness. The struggle is (not) real.

There’s a class of apps called “automated financial assistance apps”. I don’t know if that’s a good name for them, you tell me: these apps log in to your bank and credit card accounts and scour your recent transactions for subscriptions that you may not want and offers to cancel them for you.

Like…why? Who wants this? Who among us has so many subscriptions that we simply cannot keep track of them and need thousands of lines of code and privacy-invading apps to help? (Jon 1:18)

This sounds like a solution looking for a problem, but thankfully Truebill has thought long and hard about the problem it is solving:

The average American spends around 44 hours per week working. That doesn't leave much time to think about money.

Umm, no. There’s nothing that people spend more time thinking about over the course of their lives than money. So what’s the real deal?

Truebill collects, among other stuff, your “name, e-mail address, mailing address, mobile phone number, social security number, and demographic information” and depending on what you do in the app, they may also collect “your personal financial information, such as your current income level, current expenses, investable assets, and other financial information”

Ugh. OK, so what does it do with this info?

We may use your e-mail address or other personal information (a) to contact you for administrative purposes such as customer service, to address intellectual property infringement, right of privacy violations or defamation issues related to your User Content posted on the Service or (b) to send you promotional materials, offers, messages related to the Service and the activities of third parties we work with.

OK. So these apps rifle through my personal financial information under the guise of “helping” me find subscriptions that I have somehow become blind to coming off my credit card each month, and then they allow third parties to use that data to sell stuff to me. Got it. Hard pass.

When cash is gone, so is our privacy

As we become a cashless society, what we’re really giving up is some of our freedom from surveillance and control. Cash is really about citizen control or lack of control. When people pay cash for stuff, nobody else has any way to see that transaction. There’s no way to tax it, there’s no way to track the goods that transferred hands, and there’s no way to build useful statistics like GDP (said nobody ever). Credit cards, Interac, bank transfers, and whatever other tools banks whip up are far more desirable to those who wish to watch us.

Governments love centralized finance and somewhere in that love affair is the reason why cryptocurrencies will never reach fiat status. They’re fine for speculating, or even for buying the infrequent thing from that weird guy or store that takes crypto coins, but because those transactions have the same invisibility as cash, cryptocurrencies are never going to get the high-level support to become mainstream.

Daniel Jeffries has written a very long, but very interesting, treatise on just this topic. Bookmark it for your commute (do we still commute during Covid?) and give it a read.

Beheading isn’t quite bad enough

In case you were wondering exactly where the line is for Facebook to ban your account, it’s somewhere between beheading people and rushing the US Capitol.

Bannon suggested in a video posted on Nov. 5 that FBI Director Christopher Wray and government infectious diseases expert Anthony Fauci should be beheaded.

…

“We have specific rules around how many times you need to violate certain policies before we will deactivate your account completely,” Zuckerberg said. “While the offenses here, I think, came close to crossing that line, they clearly did not cross the line.”

Sorry, I got that wrong. It isn’t that beheading isn’t a bannable offense, it’s that Bannon hadn’t said enough people should be beheaded. Obviously, if Bannon had said…say… three people should be beheaded, that would be enough. Maybe. It’s not clear where that line is.

I can’t even imagine the type of thought process that goes through the minds of Facebook moderators, Zuck included. It takes a really spectacularly slanted view of the world to stand up in front of your staff and say someone hasn’t yet been violent enough to be banned. Especially when the violence in question is literally beheading someone.

I think about that all the time when I am on Facebook. I’ve left Facebook one less time than I’ve come back, and I lose a little bit more respect for myself every damn time I come back. Maybe one day I can even those tallies up.

• * *

If you’re a free subscriber and you like what you’ve read, encourage me to write more. Click the button below for an outrageously good deal!

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

The revolution will be streamed

I attended a talk given by one of my company’s board members a few years ago. Board members are usually influential decision makers from adjacent companies that can bring relevant vision to a company, but are not a direct competitor. I can’t remember the name of this person, but at one point he said that one of his biggest motivators is the fact that “you have to deal with the world as it is, not how you want it to be.”

I think about that lot because it is brilliant, but it’s not brilliant because of what it says. It’s brilliant because of what it leaves unsaid. Honestly, it sounds kind of defeatist at face value. It can be read as: “you can’t do that because the world is not that way”.

My takeaway was “if you don’t like the world, change it, then re-engage it on those terms.” That’s very motivating to me for a few reasons. First, it leaves the door open for virtually any kind of change because it implies that if the world were to change, you could do the thing you want to do. Second, it encourages long-term thinking which is in critically short supply these days.

Tactical thinking is planning three steps ahead of you. Strategic thinking is envisioning 100 steps ahead. But you can’t realize your strategy without developing shorter-term tactics to get there. Changing anything meaningful about the world is a strategy that takes time and resources, but it can be done.

That one little sentence packs a wallop, whether he meant it that way or not. I suspect he did.

• * *

Dismantling government internet control in improbable ways

We enjoy a fairly robust and unfettered internet in North America. It’s quite a bit more centralized than I like; I don’t like the fact that a (very) few companies have essentially centralized the Internet and therefore weakened its robustness, but by and large it is corporate interests, not government, that is degrading our internet here. Overseas, or “Rest of World” as we like to think of anything outside our little bubble, governments are the enemy of a decentralized internet.

The media eschews complexity, so it generally dumbs down internet control to phrases like “internet kill-switch” to illustrate the fact that the internet can be shut down in small geographies. That’s true, but it’s much harder to take down the internet than this type of journalism would have us to believe. The internet was designed to route around damage so simply taking out a router or two is usually not enough to take down the internet. It gets even more complicated when a government tries to surgically remove internet for citizens, but maintain it for themselves.

The government of Sudan learned this the hard way when it attempted to control the citizen protests that removed a generational dictatorship by disabling the internet. There’s a new government in Sudan now, so you can see how that worked out for the despots. The great part about this story is that the story involves a single person who sued their cellular provider for failing to provide internet access as promised in their service agreement. Abdelazim Hassan is a lawyer in Sudan who brought back the internet without going toe-to-toe with the government at all.

It’s hard to overstate the incongruity — the absurdity, even — of arguing the finer points of contract law in the wake of a civilian massacre, before a judge who answers to an unaccountable military regime. And yet, there’s nowhere but local courts to turn to when the government takes away the internet. There’s no international treaty protecting internet access, no global legal body that sanctions a rogue government or cellular provider. There’s just a guy like Hassan and his rather niggling complaint that someone didn’t live up to their end of a deal.

And improbably, it worked. The court ordered Zain to give him back his internet, and, on June 23, it did. But Hassan was just one plaintiff, suing on behalf of only himself, so the court and the cell phone company took a narrow course of action: Ignoring the millions of other customers Zain served, the court switched mobile internet back on … for him.

It’s worth mentioning that Rest of World has excellent journalism about parts of the world we never talk about here in North America.

The internet is populated with vegetables

I’ve long suspected that many of the people I have been interacting with for years on the internet seem too dumb to be human and may, in fact, be hyper intelligent vegetables. There is mounting evidence that I am correct, and I may have been corresponding with spinach plants this entire time.

It may sound like something out of a futuristic science fiction film, but scientists have managed to engineer spinach plants which are capable of sending emails.

Through nanotechnology, engineers at MIT in the US have transformed spinach into sensors capable of detecting explosive materials. These plants are then able to wirelessly relay this information back to the scientists.

Yup. That’s happening in the world.

The email angle is funny, but the basis of using plants to detect environmental changes is pretty sound. Because they are much more in tune with the earth and weather patterns than us car and house-bound humans, plants are pretty good at alerting us to changes before we can notice them. The problem, of course, is that they’re plants and communicate about as poorly as you’d expect. So, while email may not be the best thing to use as a message medium, more sophisticated notification systems are probably a really Good Thing ™ to build.

Smart not smart

Internet security is hard and any site can eventually be beaten if enough bad guys focus enough resources on it. This is because attackers only have to win once, whereas defenders have to win every time. Much like a casino will always win in the long term, sufficiently persistent bad guys will eventually win as well. But the infosec community isn’t utterly hapless.

The infosec sector has a good set of principles we follow when we design systems to limit the damage of a data breach. One of the oldest and easiest to implement rules is that we never store passwords in plain text that is readable. Competent developers will always hash passwords which means that if a bad guy gets the password list, they won’t be able to tell what the passwords are in “plain text”.

Well, guess who didn’t get the memo? The group of people who think they are so smart that they need a group to tell everyone how smart they are totally screwed this up. The British Mensa association was hacked recently and every member’s password was compromised because, you guess it, they were being stored unhashed and in plain text.

If only they had hired someone smart like them.

I have to give Graham Cluley props for this opening paragraph:

The website of Mensa – the club for people who have scored highly in an IQ test but who feel their social lives would be improved by hanging out with other people who chose to join a club after scoring highly in an IQ test – is said to have suffered a cyber attack.

And just to add the final touch on this story:

The discovery of an attack was swiftly followed by two of British Mensa’s board members quitting, citing concerns that the organisation is not properly protecting members’ data.

Remember, drama addicts who clutch their pearls and faint are powerless. It is much easier to effect change from within and giving up your seat at the table removes all that power. But Mensa people know that, right?

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---