I Grok. You Grok. We All Grok For ngrok!

I have been wanting to write about ngrok for a long time but it’s a tough topic. It’s not tough because it’s complicated. It’s not tough because it’s hard to explain. It’s tough because it’s so stunningly simple that it’s hard to carve out enough words to create a post about it. It’s like explaining a coat hanger; unchanged since creation; as functional now as it was in its primordial form. But, today I am going to do it. As the (para)phrase goes “don’t mistake my simplicity for simplicity”. Or, something like that.

Before we start this journey, I want to share with you the most interesting thing I learned while writing this.

Grok/ˈɡrɒk/ is a neologism coined by American writer Robert A. Heinlein for his 1961 science fiction novel [Stranger in a Strange Land](https://en.wikipedia.org/wiki/StrangerinaStrangeLand “Stranger in a Strange Land”). – Wikipedia

Mind. Blown. The term “grok” has always been a Douglas Adams thing for me. OK, now we can move on.

What’s an ngrok?

The website describes it best. Ngrok is a tool that “exposes local servers behind NATs and firewalls to the public internet over secure tunnels.” Which sounds awesome, and it is.

You install ngrok on your system, launch it by telling it what type of tunnel you’d like (usually TCP or HTTP) and ngrok responds by giving you a publicly available URL that other folks can use to connect to your local app. It is that easy.

Why would I use it?

I am a sysadmin and as such I have a pretty broad knowledge of a lot of tools. I have a deep knowledge of a subset of those tools, but the rest…well…sysadmins are tinkerers and I’m the King Tinkerer. I can get just about anything to the “it works” phase but sometimes the end spit and polish are beyond me so I frequently have a need to hit my app from outside my local machine. And I’m lazy, so I want to do that with as little work as possible.

Collaborate

For that reason, I sometimes need to test or demo code in place, rather than going through the pain of spinning up a new server for it, or pushing it to an existing internet-connected environment. With ngrok, I can provide anyone with a temporary ngrok URL to access my local system.

This is one of the great uses of ngrok, in my opinion. If I am running a little code snippet that’s not really complete, or perhaps I haven’t gotten around to creating a safe home for it on a public server, or the corporate IT guys won’t poke a hole in the firewall for me, my colleagues can just hit the temporary ngrok URL and see my service for themselves. Home users don’t even need to muck around with port forwarding on their routers.

Special mention: Mobile testing

It is also incredibly useful for testing mobile services. It is notoriously difficult to emulate the mobile experience without actually using a mobile device. The internet is polluted with services and tools to help developers emulate the mobile experience. And, while it is possible to modify the DNS of some mobile devices to hit your service, you would have to expose it on the internet to make that happen. Using ngrok, you have none of those worries because users will simply hit a normal available URL on their phone and arrive at your internal service via the ngrok tunnel.

Inspect

When ngrok is launched, it displays your assigned URLs that can be used to access your local service. That’s awesome. But, even better, it also launches a browser-based inspection tool so you can view requests and responses. By default, the inspection tool runs on your localhost on port 4040. I ran the command ./ngrok http 80 and ngrok fired up and displayed the following useful information for me.

Ngrok gives me two public URLs in the ngrok.io domain, one for plain HTTP and one for HTTPS secured with an ngrok TLS certificate. It also tells me where requests to those URLs are going to end up. In my case, on a local copy of Nginx running on port 80.

It also provides me with a Web Interface URL which is my local traffic inspector. When requests are made to those public URLs, I can see the request, my service’s response, and some other neat things.

I can view the request in a few different ways. I usually prefer the “Raw” tab and we can see that my Nginx instance just responds with a page containing the text “Hi”.

Replay (and Modify)

This is where it gets truly interesting. See that “Replay” button? Yes, it does what you think – it just replays the last request. That is useful if you need to make a modification to the local code and don’t want to reconstruct the request again.

Now, click the down arrow on that blue button.

Replay with modifications! This is very useful if you need to make changes to the request on the client-side and don’t want to have the request reissued. Clicking the Replay with Modifications button brings up the page below which allows you to alter parts of the initial request, and then replay it back into your local service.

In my opinion, the most useful part is the ability to modify the host and quickly replay a request. Yes, you can use curl with the -H switch, but sometimes you need to hit a service with an actual browser or from an app, and modifying the request is almost always easier than mucking about on the client-side. You can also click the minus button beside each parameter to remove it entirely from the request, or lick the Add Header or Add Trailer buttons to add a new parameter. You get bonus points if you already knew about the existence of HTTP Trailers, although I strongly feel they should be named HTTP Footers to more nicely correlate to the term HTTP Headers.

Advanced uses

The list of features available in ngrok that I will probably never use is nearly endless. Here are some of the top ones that grab my attention and I think could be very useful:

Custom domains and subdomains

Whenever I fire up ngrok on my free plan, I am assigned a random subdomain. That is fine for me, but I can understand why some people would chose a paid plan to allow either personal subdomains (something.ngrok.io) or their own FDQN (myapp.com). This is especially true if your ngrok URL is buried in your code. Modifying your code every time you restart ngrok would get old fast.

TCP tunnels

Ngrok doesn’t just work with HTTP. It can start up TCP tunnels on arbitrary ports to support a wide variety of scenarios.

HTTP Basic Authentication protection

Basic Auth isn’t the best, but given that ngrok should only be used for testing and development, it’s a reasonable precaution to protect your tunnel. Paid plans also support IP whitelisting to add an additional layer of protection if desired.

You grok now?

I hope this gives you enough info to investigate ngrok. I could not hope to compile a comprehensive list of its possible uses which is how I know it is a great tool. I have used it to demo website changes to my partner, configure OSSEC SIEM integrations before pushing them live, and to debug my API code while developing. You will likely find a bunch more uses that I haven’t even thought of yet.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Past Career Chronicles: Jon the Repo Man

At some point, you get far enough into a career that you can remove the earlier stuff from your resume and LinkedIn profile. My profile starts around the turn of the century, but I didn’t just start working that day. There’s lots of stuff I was doing before then, but it’s just not relevant for my current career. Many of us worked in restaurants, retail stores, delivery routes, and other entry-level jobs before starting a career. I’m no different – I had careers in the military and hospitality prior to my current career in technology. Sprinkled within those early years are ad-hoc jobs that I explored while looking for something that would light enough fire under me to call it a career. It was during those years that I ended up repossessing furniture for a chain rent to own franchise in western Canada.

• * *

The journey to the repo man was long and winding. I had worked in restaurants for years leading up to this. So long, in fact, that I was a Sous Chef at an upscale dining chain, as well as the assistant kitchen manager. That was my first career and although I liked it, kitchens are a young person’s game. The heat, the strange hours, and the physical labour of throwing big bags and bins of stuff around started to lose its appeal after about 5 years. I started looking for jobs that were less sweaty and had better hours so I could stop working weekends and evenings.

When you’re early in a career and have almost no experience in anything other than restaurants, it can be hard to find a decent job outside the industry. The only option open to most people is to start over, at the bottom, in some other industry. I was determined to do that even though I did not know what industry. I just knew that restaurants did not have enough upward mobility to carry me too much farther. I started applying for anything that looked even remotely interesting and somehow, I ended up doing telephone collections.

Collections

To this day I am not sure what attracted me to that industry. I assume it was the promise of a job with banker’s hours in a workplace bereft of stinking deep fryers and grills. But whatever the reason, that was a short-lived job.

Most jobs are far more detailed than they appear on the outside. As the old saying goes “don’t get to close to the ballet” and the collection industry is no different. The only exposure most of us have had to a collector is a voice on the phone, usually pretty rude and aggressive, demanding to speak to someone we’ve never heard of about “a legal matter.” That may seem like the beginning of the process, but it’s not. By the time a human collector is phoning people that debt is already old and the chances of recovering it are minimal.

Collections operate on what is called “the dunning cycle”. It starts with the collections agency purchasing debt. For example, if a person has defaulted on their bank credit card with a balance of, say…$15,000…the bank will try to collect that money. The bank’s own internal collections team will “dun” the debtor and do its utmost to collect that money. The bank isn’t in the business of collecting debt, so it will only pursue this for a while, and then write the $15,000 as a bad debt on their balance sheet. You may think that is the end of it, but that’s because you’re not close enough to the ballet yet.

Banks are really good at creating opportunities to make money. That $15,000 it just wrote off is one of those opportunities. The bank will sell that debt to a collections agency for pennies on the dollar. I don’t know what the industry rates actually are, but let’s say 25 cents on the dollar. That means that a collections agency pays the bank $3,750 which transfers ownership of that $15,000 debt to the collections agency. The bank has now written off the $15,000 debt and recorded $3,750 against it. It is out $11,250 and quietly leaves the room. The bank is done, but the collections agency is just getting started.

The collection agency now legally owns that debt of $15,000 that it purchased for $3,750 which leaves a lot of room for profit. As long as the collections agency manages to get that $3,750 out of the debtor, it breaks even. Anything the collections agency manages to recover above that $3,750 is free money. There’s $11,250 worth of wiggle room here and the dunning cycle kicks off in the hopes of recovering as much of it as possible.

The collections industry tries to be efficient. Automation is cheap, humans are not. When that debt is purchased, it first goes through automation. Generally, this takes the form of a series of subsequently more threatening letters being sent to the debtor’s last know address, as well as a series of robocalls to the debtor’s phone number of record. If these automated processes work and the debtor pays up, that is a huge win for the collections agency. It has recovered its debt, made a nice profit, and hasn’t spent a single human minute on it. That’s as close to free money as you can get.

However, many debtors don’t respond to automated letters and calls. After a debt has been through the automated phase of the dunning cycle without a result, it will get assigned to a human collector. All collectors are not created equal and generally, seniority is broken down by the level of debt owed. Debts of less than a few thousand dollars are usually assigned to junior collectors, whereas the big-ticket balances usually go to the more senior people. One of those collectors starts calling.

There’s an additional step that may be needed. If the human collector determines that the contact information for the debtor is not valid, then the debt can go into “skip tracing”. You can probably figure out that this is the process of “tracing” the person who as “skipped” town. Sometimes collectors do their own skip tracing, sometimes there are dedicated tracers in the agency.

Collections quickly lost its appeal to me, primarily because collectors are completely powerless voices on the phone. By the time a debt has been assigned to a collector, the damage has been done to the debtors’ credit rating so there’s not much leverage there. And, recall that the debt is really old by the time a human collector starts calling, so there is not much incentive for many debtors to pay up, especially to a (usually) rude collector. Many debtors don’t hide at all – they will pick up the phone every time you call and say “nope, not gonna pay you today, either. Have a nice day.” To make matters worse for collectors, regulations prohibit collectors from doing anything meaningful.

Collectors aren’t allowed to do anything but call and mail; they cannot hop in a car and go talk to debtors in person. Collectors are only allowed to call during certain times, are restricted in how often they can call, and the language collectors are permitted to use is heavily regulated. In short, collectors are just squawky voices on the line that can’t do anything real except be an annoying squawky voice on the line. Getting told to get lost a hundred times a day loses its appeal pretty quickly.

I decided that it would be better if I was able to actually go to people’s houses and workplaces to collect debts.

“Account Management”

I left collections and took a job as an account manager at a rent to own store. The RTO industry is huge and these days even extends into real estate. During my tenure, we were just renting out furniture and appliances. Ostensibly the RTO process is very straightforward. You come into the store and select a couch. If you pay $75 every month for 24 months then the couch is yours. If you decide you don’t want the couch, no problem. Just call us up, we will come to collect it, and you stop paying for it. Easy, right?

Not really. RTO places are unique in that they are not credit lenders and therefore do not have any of the restrictions that apply to lenders. Taking my couch example above, the customer would pay $1,800 for that couch over a two-year period. At our store, the rule of thumb was to get 3 times the value of the item so that couch would have retailed for about $600 at another furniture store. But, of course, that furniture store wants customers to pay for it, with cash or credit, before taking it home. We would give it to you for two months deposit ($150) and even deliver it. It is against the law in Canada to charge 300% interest, so a credit lender could not possibly hope to get that much interest for a couch. But RTO doesn’t lend money, so it can charge as much as it can possibly get for stuff.

Let’s get a little closer to the ballet again. Here are some of the nuances of the RTO industry:

Delivery

When I said we would deliver the couch to you, what I really meant is that we will deliver the couch to you. At the address you wrote on your rental application. We did not let customers pick up their own stuff; even small things like lamps had to be delivered by us. This policy allowed us to be sure that the customer does actually live where they said, and it allows us to be sure of where we last saw the item in case of problems later.

References

Because we were not extending credit, we did not run credit checks. That may seem foolish, but honestly – very few RTO customers have good credit. If they did, they would not be entertaining the idea of paying 3 times over for a couch. What we were more concerned about is our future ability to recover the item. Therefore, in lieu of a credit check, we asked for references. That was unpopular because nobody wants a rent to own sales guy phoning their buddy or employer to verify details about them. But, most of our customers were not in a strong bargaining position so that is how we went about our business.

At this point, we should have everything we need to close this deal. We know the customer, we’ve confirmed the information on their application so we have reasonable grounds to think this person is legit. Further, we have collected 2 months of rent on the item and lastly, we’re going to drive to their house and deliver the item for them which again confirms their address and living situation. Lastly, this customer knows that if they ever do not want the couch anymore, just call us and we’ll pick it up and they stop paying for it. That’s a pretty good situation. What could go wrong?

Repo

I knew going into this position that some customers are not going to pay and are going to try to keep the items. I didn’t mind the prospect of that part of the job because it filled that void that collections could not. Because the items are owned by the RTO store until the last payment is made, we can, and did, go to our delinquent customers’ houses to recover our stuff. It was more satisfying than being a powerless squawk on the phone. But, I was not prepared for how much of this new position would consist of repo. What I thought would be a fairly rare activity turned out to be 80% of the job.

There were four of us working at the RTO store – the manager and three account managers. Us account managers were paid a modest wage and were expected to make most of our money from bonuses. Monthly bonuses were triggered by many things, but the one we chased the most was the “100% paid-up” bonus. I can’t remember how much that bonus was, but it was enough that we were always trying to get it. You can probably guess from the name that bonus is triggered when an account manager has zero delinquent accounts on the last day of the month. We almost never got that one, but as the month went on our collections activities would rachet up to a feverish pace near the end in an attempt to make it. If one of us was close and the rest of us were not, we’d abandon our own collection activities to help the guy that was close. Some months we’d have all three vans on the road just hitting one account manager’s customers to help him get his bonus. Then, on the first of the month, it would start all over again.

There comes a time when a customer is no longer delinquent and is just not paying anymore. In those cases, we would have to go and pick up the items. Technically, it was not repossession because legally we had not ever relinquished possession – recall that the item remains the property of the RTO store until the last payment is made. But that is a legal nuance which doesn’t mean a hill of beans when you’re trying to take something back from someone who doesn’t want you to have it.

In theory, because the RTO store still owned the items, we could show up with a sheriff and take it back. In practice, much like banks don’t want to be in the collections business, we did not want to be in the repo business so we rarely did that. And, because of our exorbitant fees, by the time the customer stopped paying, we had usually been paid around the value of the item anyhow. But, we would still be penalized in our bonuses if we had delinquent accounts so account managers were stuck in this gray area where we had to find ways to cajole and convince the delinquent customers to let us pick up our stuff or bring their account up to date.

That Damn Dryer

This is the period of time when things got strange. Because we did not go the sheriff route, we had no authority to enter our customers’ houses without their permission. A lot of our customers knew this, and they would happily answer the door when we showed up and tell us to our face that they’re not paying and no, we can’t come in to get the thing. I remember one customer who “rented” a dryer from us. He had only ever made a single payment and it was years before I started working there. One of my fellow account managers inherited this customer into this list, and because of that, he had never made 100% bonus.

We’d go to his house periodically, hoping to find some scenario where we would be allowed to get the dryer. Maybe we were just hoping the dryer would break down so he’d willingly give it back to us, I am not sure. But I do know that this guy was so savvy about the laws that he didn’t even care if his kids answered the door for us when he was not home. I remember his teenaged daughter answering the door, but even if she said I could come in, it would not be legal because a minor can’t give that consent. It was a little frustrating, to say the least.

We became obsessed with this guy and his old outdated dryer. Finally one day, my store manager gathered us together and told us tomorrow morning was the day we’re getting that damn dryer back. We met at the store very early the next morning. Each of us got into a separate vehicle and we drove to this guy’s house. He parked on the street, and we parked (legally) in front and behind him. Maybe a tad too close, who’s to say? But, he was unable to get his truck out to go to work.

After a few hours standoff and hurling many threats at us, he finally threw the door open, called us one last really bad name, and then told us to come in to get our dryer.

My colleague got his 100% bonus this month for the first time in his career.

That one turned out well, but for each one of those, there’s a bunch that did not. In the end, the job just wore me down because it was too full of sadness and anger. With one exception, picking up our gear was never a happy affair. Customers were either sad that they could no longer afford a dining room table, or they were mad that we were taking their dining room table away.

The Aristocrat

On the other end of the customer continuum, we had a woman who took the rent to own concept in completely the other direction. This customer is the exception I mentioned in the last section.

She had rented an entire apartment’s worth of stuff from us, right down to the paintings on the walls and the lamps on the (also rented) tables. She paid on time, every time, and was a delightful customer.

She also changed every single stick of furniture in her apartment every few months.

She would come to the store, pick out an entirely new set of furniture from beds to couches and everything in between. She would pay her two months deposit and we would dutifully go to her apartment, remove all our existing stuff, and bring in her new stuff. We all knew each other so well that she would give us her key many times so we could do the swap out while she was at work.

Despite the fact that we always presented our service this way: “you’re just renting it but if you rent it for 24 months, it’s yours”, she was the only customer we had that really internalized that and had fun with it. She enjoyed coming home to a totally new apartment periodically. Who wouldn’t?

The bed incident

After many months of running around town arguing with people, creeping around their houses, peering in windows, and generally just being stressed out by all the crappiness around me, I had one final experience which ended my repo career.

The “bed incident” involved a customer who had a good payment record. He had rented 4 beds from us a few months ago, and out of the blue, he called us to pick them up. We generally don’t pry too much because, honestly, we’re just happy to either get paid or get our stuff back most of the time. One of the other account managers and I hopped in our truck and headed out.

The customer met us at the door and let us go upstairs to the bedrooms. It was a two-story townhome and there was almost no furniture in it. It was not dirty or damaged, it was just obvious that these people did not have much. We started taking the beds apart and loading up the truck. While bringing one of the last loads downstairs, I caught a glimpse of the small kids in the kitchen. Presumably, these were the users of these beds and they were looking at me crying while mom tried to shush them.

I remember realizing “Jesus…I am taking these kid’s beds away.”

Most of our customers were not nice people. Most of them either deliberately tried to steal from us and had no intention of ever paying us a dime past the deposit, or they started out fine but at some point decided we were the enemy and then decided to just stop paying us and steal our stuff. It’s really easy to get jaded and think they’re all like that. But the truth is that there were a bunch of customers that I never even met. They silently paid on time, their names were automatically checked off the list in the sales system, and I never even knew their names.

This was one of the customers who had just hit hard times. He was not trying to rip us off – he called us to pick up the beds before he was delinquent. And here I was taking a bunch of kids’ beds away.

I left the job not too long after the bed incident, and that was the end of my collections/repo career. It was an incredibly dynamic time of my life – you can’t go to work sleepy when all your customers are trying to outfox you. But, it also subjected me to a lot of things that I did not want in my life. Chief among those was the industry I was working in. Much like “payday loan” places, it seems the RTO industry disproportionately targets vulnerable people which wasn’t something I wanted to be a part of.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Internet Companies Behaving Badly

Fair warning: this one is a bit of a ramble. I have a lot of thoughts on the current state of the consumer internet, but depressingly few solutions.

• * *

The internet is one of mankind’s greatest achievements. This global communications network underpins almost every aspect of life in modern countries. Non-technical people think the internet means Facebook and email, but the internet is much bigger than the parts we can see. It facilitates every facet of our daily lives; quietly passing data at the speed of light to keep our cars running, our houses warm, our hospitals lit, and our paycheques flowing. It is also almost entirely unregulated which means everyone has equal influence. That sounds great on paper, but is that the type of internet we really want?

While much fanfare was made when Apple broke the trillion-dollar market capital ceiling in 2018, it was far from the first company to do so. The first company to break the $1T market cap was a Chinese oil and gas company named PetroChina which passed $1T 12 years prior in 2007. It would take another 11 years before Apple became the next $1T company. The floodgates then opened as Amazon, Microsoft, and Alphabet (Google’s parent company) followed suit in short order. A trillion-dollar company isn’t that surprising anymore, but it is interesting that the only trillion-dollar companies that aren’t in the very rich O&G field are all technology companies.

Two of those tech companies depend entirely on the internet for their existence. Amazon would have no storefront and no way to tap the global market without the internet. Google makes the bulk of its fortune from the ads it serves to internet searches; without the internet, there would be no searchers. While Microsoft and Apple depend heavily on the internet for their app store and Azure offerings respectively, they still have a foot in the good old product business. They build and sell physical stuff and useful services in addition to their internet revenue streams. Amazon and Google do not have any similar product lines and the existence of the internet is an existential requirement for those companies.

Companies can be good stewards of things, but it is rare, especially in America. The United States government is overwhelmingly controlled by corporations which is a situation to be expected by the world’s prototypical capitalist country. Consequently, the American internet is already fragmented and less useful than it is elsewhere. Communications companies use tactics such as “zero-rating” to leverage users into visiting sites the company is being paid to promote and thus drives less traffic to non-paying competitor sites.

• * *

Companies like Google own the internet through the large portion of the browser market that Chrome dominates. Google wields the power of Chrome’s large user base in the corners of the internet that most people do not know to exist. Yet these places completely control how the internet functions. Places such as the SSL CA/B forum where Google holds enormous power over seemingly mundane things such as how secure certificate information is displayed in the browser address bar, and how long SSL certificates should be valid before requiring (paying) renewals. The balance of power in the CA/B forum lies with the browsers and the relationship in those forums is regarded as unbalanced and toxic by some industry pundits. While the nature of the decisions made in the CA/B forum may not seem to be of interest to end-users, be assured that end users pay for these decisions, usually out of their pockets.

In the same predatory vein, Google has also recently decided to stop allowing third-party cookies in the Chrome browser. This is a very good thing for internet users because third-party cookies are a perversion of the original cookie standard and provide very real tracking and surveillance capabilities to advertising companies. Google states that this initiative is combined with the rise of its Privacy Sandbox project which allows advertisers to still put relevant ads in front of people, but requires less private data to identify those users. In practice, we cannot ignore the value of advertising within Google, so it can’t be ignorant to the fact that removing third-party cookie support will deal a killing blow to many smaller internet advertising companies. I think the truth is that Google’s long-game to penetrate the browser market with Chrome has allowed it to collect much more data on internet users than any other internet advertiser. Thus, Google no longer needs third-party cookies to put relevant ads in front of users, so it is shutting off that lifeblood for other internet advertisers which will effectively bankrupt a lot of them. It doesn’t take an MBA to figure out where Google thinks those displaced advertising customers are likely to end up.

Amazon has its own share of bad behaviour stories. When Jeff Bezos decided he wanted to buy Diapers.com, he used Amazon’s tremendous technological advantage to show Diapers.com that Amazon could wipe them out in a hot second if they did not take the offer on the table. I don’t mean Bezos showed the Diapers people a few power points. I mean he built a system to track the prices of products on Diapers.com in real-time and lowered Amazon prices below the Diapers’ prices every minute of every day. At one point, Amazon was tracking to lose $100 million dollars in the diaper category because of the pricing it set to beat Diapers.com. Amazon was literally prepared to lose $100 million dollars just to acquire diapers.com. That is not the fair gamesmanship we desire in the business world. That is not an example of one company simply being better than the other. That is an example of a company that has enough power to influence the behaviour of an entire country of consumers. And for no other reason other than to make more money.

Apple isn’t exactly blameless, but, arguably, it has “honourable” intentions behind its missteps Apple alone among 1T club does not engage in advertising. Amazon advertises to get customers to its site, Google lives almost entirely off advertising revenue, and Microsoft’s Search Network is an advertising revenue stream. When a company sells ads, it necessarily has to harvest user demographics and behavioural data – there is just no other way to be competitive anymore. This quest to hoard the most data on the most people is what drives companies like Amazon, Google, and Microsoft into questionable ethical behaviour.

Apple collects user demographics and behaviour data as well. The difference is that Apple has not built a multi-billion dollar empire off of selling that data. Rather, it uses that data for its own use and has built a $1 trillion-dollar company on the age-old tactic of just building stuff that people want and are willing to pay for. Apple alone in the $1T tech club makes its money the old fashioned way. However, Apple has had problems in other areas. It has made unfavourable decisions regarding censorship of regions it operates in and also has some negative press about the labour practices of its primary contractor, Foxconn in China.

In America, power stems from wealth. These companies have so much wealth that they can’t be constrained by the systems we’ve built to date. When Amazon can decide to throw $100 million dollars into the firepit just to shut down a single competitor in only one of Amazon's thousands of product spaces, it showed us that consumers have no say in how the market operates anymore. The lesson is that consumers can no longer “vote with their wallet” and chose not to buy a certain product or boycott a certain company. That company can now just completely rewrite the market from scratch and remove all competing sources for that product, leaving consumers with only one choice.

Historically, successful companies meant successful regions and several decades ago that was still true. In those days, a successful company meant jobs for those located around it, and healthy donations-by-tax to keep the neighbourhood infrastructure healthy. That was a win-win, but these days, those living in the shadow of these tech company buildings are very much on the wrong side of a win-lose scenario. Successful companies these days outsource their work so there are no jobs for the local people. They also go through vastly complicated exercises each year to determine which countries to route their revenue through so that they do not have to pay taxes to the local or regional governments where their workplaces are. These days, a successful company is a parasite on the populace around it. It consumes resources and erodes infrastructure while striving to give as little of its money as possible back to the community. For some reason, America likes that. We know that because it awards these companies with trillion-dollar market caps.

• * *

Those that do not like the current market situation – which is everyone but the extremely wealthy – frequently turn to government regulation as the solution. Elizabeth Warren is currently campaigning for the 2020 U.S. Presidential election on a platform that includes breaking up the big tech companies. Her view is that breaking up these internet giants would weaken their power and restore the consumer voice in the market place. I think that the truth is that those smaller companies would form cabals and join forces anyhow, and the net result would remain much the same for consumers.

Another option on the table is to leave the companies to their own governance, but strengthen regulations and increase fines for bad behaviour. In theory, a company that operates without due regard for regulations would become non-profitable because it is wasting so much money on fines and litigation. But, there comes a time when a company has become so large that it can afford any fine as a cost of doing business, and even negotiate that fine in advance to facilitate budgeting. When Facebook negotiated its own $5 billion fine for bad behaviour, its stock surged. Facebook stated months earlier it had budgeted for $3B but apparently $5B is close enough for the pools Facebook plays in. When a company can simply budget away the GDP of more than 30 countries in the world as a line item, what chance do fines have of hampering this behaviour? And, let’s not forget, Facebook isn’t even anywhere near being in the $1T club at this point in time and it can easily afford this fine.

I believe there is no way to stop what is happening now, but there is a way to prepare a safe landing place for the future. The consumer internet is firmly in the hands of a very few very powerful companies, and nobody has the ability to outfox these companies into weaker positions. I believe that the MAGA era (Microsoft, Amazon, Google, Apple – what did you think I was referring to?) is upon us and will run its course. Much like the grandiose shopping malls of yesteryear surged into dominance and then faded into husks dotting the landscape, so shall these first fumbling attempts at global consumer dominance wane and the marketplace will return to something more similar to the bazaar that we’ve become accustomed to.

However, a pre-requisite for re-stabilization of the market at a later date is the existence of a relatively level playing field on internet speeds. The practise of zero-rating in America is very dangerous. That type of behaviour removes the level playing field of the internet and strongly encourages consumers to use only the sites that are excluded from their internet bandwidth caps. That means smaller companies in the same space will wither away and no longer exist when we need them most. Without creating a stable safe place for everyone to access the internet in the same way we access electricity now, we will not be able to return to a post-shopping center existence. There will be nobody left for us to return to. The creation and enforcement of net neutrality, an internet where no company can purchase special access, is the only tool that will allow the current internet to be usable in later years.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Everyone gets their own personal newspaper

“Happiness is a choice”. I have a friend who always says that. His son passed away which is something no parent should have to endure, so he knows what he's talking about. It sounds like a trite platitude but I've found it's actually true for me. While some people have legitimate mental health issues, and depression is a clinical condition that can need professional help, sometimes just deliberately trying to be happy can pull you out of a downward spiral. At least, it works for me sometimes.

Hacking highschool

Do you know about those devices that governments can buy to crack open iPhones and other encrypted devices? Well, in a spectacularly accurate example of why encryption back doors won't work, this technology has become mainstream and is now widely available to pretty much anyone who wants to buy it including school administrators.

These tools are basically weapons, encryption is specifically named as a weapon in U.S. export controls. They circumvent security measures which is illegal in almost every scenario in the U.S. and many other countries. But law enforcement has always been able to break the law for the greater good. That makes sense, but what chills me is that these tools make their way down the para-government chain over time and become widely available to everyone. Consider police officers 40 years ago; they had a pistol and handcuffs. Contrast that with today’s police officers that show up on the scene in amoured personnel carriers, drones in the air, and all manner of automatic weapons and hand cannons. And today, what was once a powerful tool designed for use by border agencies and other police forces, is now available to the petty fiefdoms of school administrators. I wish governments were a little more careful about what genies they let out what bottles, because the genies that erode our personal safety and privacy can’t be put back in their bottle.

Everyone gets their own personal newspaper

I’ve written in the past about some of the ways in which social media skews our view on the world by feeding us “news” determined by algorithms that aren’t really interested in giving us the news; they’re interested in keeping us on the site as long as possible. On a recent TWiT (This Week in Tech) episode, Ashley Esqueda said something interesting like “remember the idea of everyone in the world getting their own personal newspaper?” which is pretty much the case these days due to these algorithms feeding us each unique content. Sure it’s a nice idea to get personalized news you care about, but the current situation is far past that, and we’re deep into the “no two people get the same news” now. This results in a polarized society full of people who actually have no real clue what is going on, despite their good intentions to keep up on “the news”. Because the dominant social networks are located in the capitalist-crazy U.S., I don’t see any future where they are reigned in, but the good news is that Facebook and Google are both facing very wide-ranging anti-trust lawsuits these days. Hopefully, they will be de-platformed and lose some of the power that they’re currently wielding against the world’s best interests.

A human is worth about $1.4 million dollars

Boeing is being fined $2.5 billion dollars for multiple 737 Max crashes that killed 346 people.

“Boeing's employees chose the path of profit over candor by concealing material information from the FAA concerning the operation of its 737 Max airplane and engaging in an effort to cover up their deception,” Acting Assistant Attorney General David Burns of the Justice Department's Criminal Division, wrote in a statement.

And that is it. No criminal charges. Just money.

About $500 million of that money is earmarked for the victims’ families and the vast majority of the rest is to compensate companies for the money they lost by having to ground their 737s while the investigation ensued. With a market cap of $120 billion as of today, the fine will barely be felt through the company, and once again, capitalism wins over basic human decency.

If you liked this newsletter, there’s more to be had! Subscribe to get additional, subscriber-only, editions sent directly to your inbox. Here’s a nice button with an outrageously good offer behind it!

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Custom OSSEC Integrations

Note : I mistakenly sent this post to only paying subscribers yesterday, but it was supposed to be for all subscribers. I apologize to those of you who are getting this twice.

• * *

OSSEC is a popular Host Intrusion Detection System (HIDS). It is very capable out of the box at notifying system administrators of indicators of compromise such as suspiciously changed files and taking action against badly behaving IP addresses that are doing nasty things like credential stuffing. OSSEC doesn’t need a lot of fancy configuring – its basic functions just work after install, and most of the fun comes from configuring it. This article isn’t a deep dive into OSSEC, but it is an overview of one of my favourite features: OSSEC custom integrations.

Daniel Cid is the original author of OSSEC and maintains his own OSSEC fork and instruction here. Once you have OSSEC installed following those instructions, continue on for some fun.

Custom integration support

You should find OSSEC in /var/ossec. We’re going to pay attention to two things within that directory:

• /var/ossec/etc/ossec.conf file, and

• /var/ossec/integrations/ directory

The custom integration feature of OSSEC is a simple, but really powerful feature, that allows system administrators to cause arbitrary things to happen in response to an OSSEC action. Primarily, I use custom integrations to send OSSEC block data to customer SIEMs such as Splunk. However, a custom integration can call any executable file so the possibilities are literally endless.

The basic custom integration script

Integrations are executable files and the basic criteria for an integration to fire upon an OSSEC event are:

• Each integration is associated with a log file that OSSEC is monitoring

• The integration file must live in /var/ossec/integrations/

• The integration file must be named custom-$FOO, where $FOO is arbitrary.

• The integration file must be configured in the /var/ossec/etc/ossec.conf file

Let’s look at at a shell script that will send data to an HTTP endpoint:

#!/bin/sh ALERTFILE=$1 APIKEY=$2 WEBHOOK=$3 # Send it curl —data “$alertlog” “$WEBHOOK”

OSSEC sends the log file, any configured API key, and the endpoint to the script. We capture those in the ALERTFILE, APIKEY, and WEBHOOK variables, but that is not necessary. It just makes the script easier to understand.

OSSEC also populates a variable named $alertlog which contains the actual log entry that triggered the alert.

If you’re sending data to and endpoint that needs authorization, use the APIKEY. For example, the curl call would look like this if the endpoint was a Splunk HTTP Event Collector (HEC):

curl -H “Authorization: Splunk $APIKEY” —data “$alertlog” “$WEBHOOK”

Obviously, you would want to include some error checking and logging into this script. You can also manipulate the heck out of the content of that $alertlog variable to customize the payload or extract/inject interesting data. I have a few integrations where I have added data, such as GEO IP data, to the payloads before sending them out. But, these are the bare bones you'd need in order to send data to some endpoint, such as a Splunk HEC.

Triggering the custom integration

Because the integration is just a script or some other executable file, you can build it and test it just like any other piece of software. However, once you’re happy with it, just plunking it into the /var/ossec/integrations/ directory isn’t going to work. You still need to tell OSSEC about it.

If the script above is saved in /var/ossec/integrations/custom-basic_script, then we would need to configure the /var/ossec/etc/ossec.conf file with a section like this:

custom-basicscript <eventlocation>jonwatson.ca. 45TERT$GU)JHTE https://my_splunk.instance/services/collector

This configuration tells OSSEC to fire the /var/ossec/integrations/custom-basic_script file whenever an event is logged about the jonwatson.ca site in the OSSEC log.

The event will be sent to https://my_splunk.instance/services/collectorusing the 45TERT$GU)JHTE<API key if I use the second curl example which utilizes the -H Authorization header.

Other ideas

I primarily use the custom integrations to send data to SIEMs, but there are others. There are PagerDuty and Slack integration scripts that do other things under different criteria. That is how we get OSSEC to let us know of more critical things it is seeing. There are a million things you can do with these custom integrations, even kicking off a runbook or series of events. If you have any interesting OSSEC integrations running, I’d like to hear about them in the post comment.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Useless Skills I Learned In The Navy

courtesy of the author. it’s me!

I was in the Navy when 9/11 occurred. I had recently finished my initial trade training and overseas I went. I released when I came back, for reasons which would need a whole separate story, but 17 years later there are still some skills I can’t seem to forget.

In no particular order, here are the things taking up space in my head that I wish I could use for other things.

• * *

Semaphore

Ah, semaphore. Making communicators the Hula Girls of the sea for hundreds of years.

Morse Code was, and perhaps still is, a very useful tool. Most of its utility comes from the fact that it can be sent in many different ways. Those little dits and dahs can be sent over radio as audible sounds or sent visually through many means such as flashing light or semaphore.

The Navy still used visual semaphore during my tenure, although it’s not clear to me why. I have no idea if it is still in use, but there’s nothing quite as disheartening as sending semaphore to a nearby ship knowing full well there are a number of radios and other types of encrypted communications equipment currently doing the job much faster. Communication pundits will tell you that semaphore still has its place because the tactical situation may require silent communications, but you can’t hide a 350-foot war ship on the water, so it’s pretty hard to hide the flashing lights and flags coming from it.

The best part of sending semaphore was sunning on the upper decks waiting for something to happen. Seriously. Check the pic.

I haven’t sent or received semaphore since leaving the Navy. It comes up surprisingly infrequently in civilian life. But I still know every letter. I’d prefer my brain dump that bit of knowledge and store something more useful like my family’s birthdays.

• * *

The Watch and Station Bill

A war ship isn’t a recreational sail boat. It runs 24/7 and has to maintain readiness every minute of those hours. My ship had a crew of about 300 including the air detachment that flew and maintained the helicopters.

There’s a never ending list of duties on a ship. At the top level, we all need to do our actual jobs. We need to show up for work at our particular spot and work just like everyone else in the world. For me, my spot was the Communication Control Room or the Bridge — communicators work in both places. For others, their spot may be the kitchen or the operations room or the forward sonar instrument space, or some other space no human should ever be crammed in to. But that’s just the tip of the iceberg.

There’s a lot of duties to be done that aren’t anyone’s job. Sure, the cooks cook and the clerks clerk, but there’s nobody on-board who’s job is to clean the ship, recycle the garbage, hump loads of food and ammo around the ship and fight fires (OK, there are fire-fighters but just a few to tell the rest of us what to do). Everybody does those jobs in their “off” time. Almost every waking moment is spent doing something which may or may not have any relationship to your job.

And those are just the jobs you are forced to do. You still have to somehow carve out some time for personal stuff such as your laundry, hair cuts, and talking/emailing people back home. None of that stuff is factored into your work day.

If everyone is running everywhere to do stuff all the time, how do people get to work at the right time and place? Enter the monstrosity that only the Navy could create: the dreaded Watch and Station Bill.

There are a number of watch rotations with simple names like “1 in 4”, “1 in 3 modified” or, the worst in my opinion, the “1 in 2”. Operations folks, like communicators, can live in the 1 in 2 rotation for months.

Basically, the name means you’re working 1 in every X watch (“watches” are “shifts”, you civvie!). How long is a watch? That depends on the rotation. In a 1 in 2, the watches are 7 hours, 7 hours, 5 hours, and 5 hours which together comprise a 24 hour day. That’s a long day and it goes on for months non-stop, but it’s easy to understand. To become completely confused, the 1 in 4 is here to help.

Following the same formula, the 1 in 4 means you work 1 in every 4 watches. That sounds better, right? Wait for it.

The watches are:

0000-0400: Mid(dle Watch) 0400–0800: Morning 0800–1200: Forenoon 1200–1600: Afternoon 1600–1800: First dog 1800–2000: Last (or Second) dog 2000-0000: First (or Evening)

You’d think this means that there’s a lot of time off. But this is the Navy so you’d be wrong. During a 1 in 4 rotation everyone ALSO works 0800–1600 in addition to their watches. If one of your watches falls within 0800–1600 great, if it doesn’t then too bad for you.

Example of a 2-day rotation: On day 1, I work the Mids, off for 4 hours and report to work at 0800 with everyone else until 1600. I then immediately go back to work at 1600 for two hours, but then I am off until 0400. That’s not too bad — I get a solid 10 hours off. That’s called “All Nighters”. Navy folk are clever like that.

On day 2, I am on watch from 0400–0800 then I have to work all day from 0800–1600 with everyone else. I get 2 hours off, then I have to report for the second dog watch and that makes for a long day. But, then bliss sets in. I don’t have to report to watch until 0800 the next morning which gives me 12 hours to recuperate. That’s called “Super All Nighters”. See? Clever.

Days 3 and 4 are more crappy and then the rotation starts all over again.

My point isn’t to instruct you on how a Naval watch and station bill works. My point is that all this stuff is still stuck in my head almost 20 years after I last had a use for it. That’s crazy.

• * *

The bowline knot

courtesy Wikimedia Commons public domain

I joined the Navy later in life than your typical recruit — mostly because I was busy being a Militia MP and going to college for years prior. When I made the change to Regular Force Navy I was just barely in my 30s and had somehow made it through life happily tying things up with infantile square knots just fine. Put your finger right there to hold this while I tie the next knot, thank you very much.

Initial military training has three basic phases for every new recruit. First, the much publicized Basic Training (“Boot Camp” for the Americans among us) of which movies and novels galore have been made. The next phase is Environmental Training which is the baseline training for your particular element (sea, air, ground). Lastly comes Trade Training which is where you learn how to do your actual specific job.

Army environmental training consists of a lot of guns and digging. Navy environmental training consisted of a lot of terminology and — you guessed it — knots. Nobody has any idea what Air Force environmental training looks like.I guess how to clean crumbs off the ready-room chaise lounges? But I digress.

I was destined to become a communicator and would have little use for knots. But, I learned the 3 or 4 knots required of me and then promptly forgot them all once I left the sea training school. All but one. That damn bowline haunts me today.

I don’t care how cool you are, everyone loves lassos. — Jon Watson

The bowline knot fascinated me. With a bowline, you can make any line into a lasso. I don’t care how cool you are, everyone loves lassos. I always knew when someone was trying to remember how to tie one. I’d overhear muttering about the “rabbit runs around the tree then into the hole”…etc. I don’t actually remember the mnemonic any more because my hands just know how to do it now. It’s burned into RAM, as my computer nerd friends would say.

I’ve been out of the Navy for almost a generation at this point; there are Petty Officers serving now that weren’t even in the Navy when I left. And my house is littered with unnecessary bowline knots. I bundle up recycling with it, I make temporary leashes with it. I tie down my hatch back with it during long load hauls. None of that is new; the ol’ childish square knot used to work just as well, but now I’m a bowline guy for no particular reason.

• * *

Careers are usually built on learning progressively more complex skills and building on them. Sadly, your brain doesn’t know what a career is and it just remembers what it wants to remember for reasons that aren’t well understood yet. We don’t have a lot of control over what it decides to store for the long term. This is why you can remember things like your high school locker combination, but not your current car’s license plate number. It’s frustrating and incredibly interesting at the same time.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Who Cares About 5G Mobile Networks?

There’s a lot of hype about the new 5G mobile networks being deployed across the globe. Marketers are telling us that the internet will become blazingly fast on our mobile devices, but whatever. They tell us that every time and while it’s true, our mobile devices are generally fast enough these days, so there’s a lot of eye-rolling from consumers around this “next big thing”. I understand that, and to some extent, 5G is just the next 4G. But there are some significant things about 5G that actually do make it a lot more interesting than just more speed. Let’s take a high-level view of it so you can decide for yourself if you want to take the leap when 5G networks come to your region.

The Letter G

Freemasons aren’t the only people obsessed with the letter G. Before we can go too deep into this topic, we need to clear up this “G” thing. Many of us see WiFi networks with the name “5G” in them. Our home WiFi routers usually have a “5G” network, most public WiFi access points do by now, and ISPs are now supplying WiFi modems with their service which usually have a “5G” option as well. But that, dear reader, is not the same 5G as the mobile 5G everyone is excited about.

The WiFi “G” stands for Gigahertz which is a measure of radio frequency. The mobile network “G” stands for Generation , meaning 5G is the fifth significant change to mobile networks since their inception in the early 1970s. Yes. Mobile phones have been around since 1970. Mind. Blown.

I won’t take you too deeply into this rabbit hole, but here’s the quick tour. The 1G networks were analog only which means only voice phone calls were possible. The 2G networks were digital and ushered in the era of mobile data, albeit in a very limited and slow way. The 2G networks introduced text messaging, for example, which became a mainstay of human expression and unfortunately also gave Jack Dorsey the idea to create Twitter. The 3G networks increased the speed of 2G networks and they also supported the concept of roaming, so mobile phone users could theoretically use other company’s networks while traveling. In reality, that roaming “feature” became a cash cow for mobile phone companies so most consumers shied away from the extravagant roaming fees and just shut their phones off when they were not in their home area. Roaming fees are mostly gone now, but people of a certain age still cringe when they hear the word “roaming”. 4G was a significant change in mobile networks. It has a variety of different flavors, but most of the world is standardized on 4G LTE these days, which is completely digital and handles all of our data and voice activities over the internet using the IP protocol instead of analog noise.

Unlike the older 1G and 2G networks, 3G networks still exist. Rural folks like me will occasionally see our phone change from “LTE” to “3G” in the status bar. As you’ve figured out by now, the lower the “G”eneration number, the older the technology. The older the technology, the lower the frequency it operates at. The rules of thumb for frequency and data are:

• lower frequency means slower data

• lower frequency travels better so fewer towers are needed for coverage

• higher frequency means faster data

• higher frequency travels shorter and has trouble penetrating things like walls, so more towers are needed for coverage

These rules are true of all radio, not just the frequencies used for mobile networks. When I was in the Canadian Navy, we had a saying in the communications room: “HF (high frequency) radio travels the world”. Our primary source of information from the west coast of Canada to the Gulf of Oman was over HF radio running between 3 and 30 Megahertz (MHz), which is not “high” frequency at all in the scope of the Gigahertz (GHz) frequencies we use today, but changing its name would cause far too much confusion so that’s what we’re stuck with.

How fast is 5G?

The answer is like most things in life: it depends. 5G has three bands within it, up to a whopping 52GHz frequency. For comparison, 4G tops out at 6GHz and if we refer back to our rules of thumb, we can see that the higher the frequency, the faster the data. 5G definitely has the frequency bands to deliver exponentially faster data. But, because of the rule of thumb about coverage, mobile carriers would have to deploy hundreds of thousands of 5G transceivers in order to provide 5G coverage of those speeds coast to coast. As I look at the decade-old “3G” in my phone’s status bar, I sadly realize that there’s no way mobile carriers are going to shell out that kind of cash for those kinds of speeds coast to coast.

5G is broken down into two ranges named FR1 (450 MHz – 6 GHz) and FR2 (24 GHz – 52 GHz). F1 operates under 6GHz which is the same bands as 4G. However, even 5G F1 will be faster than the existing 4G but only by a bit, and that speed increase is mostly attributed to more efficient antennas than more bandwidth. F2, on the other hand, will be much faster but more expensive to deploy. Technically, 5G gets speeds up to 20 Gigabits per second whereas 4G averages around 20-30 Megabits per second, or 0.20 – 0.30 Gigabits. That’s a massive difference, but we’re only going to see those types of speeds where 5G F2 is rolled out completely.

When do we get 5G?

These days, 5G is available in a lot of markets. It is rolling out in Toronto, Montreal, Calgary, Edmonton, and Vancouver and those are markets where it makes sense to implement top-speed F2 stuff. The rest of us will end up with mid-range 5G in the lower F2 bands which will still be faster than 4G but may not be worth rushing to upgrade. Technology like this leapfrogs itself – meaning the devices and the infrastructure pass each other cyclically in increasingly larger scales to combat the “chicken and the egg” problem.

Take electric cars as an example. That is a classic “chicken and egg” problem. These technologies are created by private industry which has a need to continue to make money in order to survive. Therefore, a company can’t just build a bunch of electric cars in isolation and then put them in a lot and expect people to buy them. Nobody is going to buy an electric car if there is no way to charge it on the highway to go on longer trips. And other companies aren’t going to spend billions adding electric charging stations to their gas stations if nobody is driving electric cars in order to pay for them. So, we get the incremental leapfrog where a small number of cars come out and are sent to high-density markets to sell, such as large cities. At the same time, large national companies like Petro-Canada put charging stations in some of their gas stations, again on high-density roads such as the Trans Canada Highway where they will get the most use (AKA, make the most money on their investment). Those first steps are enough to get electric cars on the roads, get normal citizens driving and talking about them and wanting to buy them. And that’s how you create a market from nothing at all.

This multi-pronged approach is an effective way to jump-start a nascent industry and moving to 5G cellular is no different. Because 4G phones cannot be used on 5G networks, everyone will eventually have to buy a new cell phone and handset manufacturers are putting 5G phones on the market now. But until the towers are all upgraded, the 5G phones can’t be used to their full potential, as they will just fall back to 4G when 5G is not available. What we’re seeing now is the “tower and phone” incremental leapfrog that I just described using electric cars. We’ll get there, but it’s not an all-or-nothing proposition overnight.

The fastest 5G deployments will be best served in areas of super high density which is why we’re seeing it being rolled out in busy areas of major urban centers. Even denser areas such as arenas, shopping malls, and airports are prime candidates for 5G because a super-fast F2 5G tower has a range of less than a kilometer. This means we’re not going to see those speeds any time soon in rural areas where there can be kilometers between individual humans that can be billed for the service.

Why do we need 5G?

I know that most of you reading this are still thinking about your cell phone. “Why do I need my cell phone to be faster? My games and movies and video calls work just fine.” You’re right, but you’re thinking too small. I agree that my cell phone is fast enough, but 5G isn’t about cell phones.

We draw a line in our heads between “internet” and “mobile internet”. We chat and email and look up some things on our phones, but for “real work” we wait until we’re at home with our laptop or desktop. 5G will shatter that line because it is the technology that will enable our laptops and iPads and whatever else we have to just work connected to the internet wherever we are and we’ll stop thinking about being on the internet as a separate activity. 5G isn’t just about speed, it’s about capacity in the sense that many, many more devices can be connected to any given tower than 4G. We will no longer have to tether our laptops to our cell phones to get data, the laptop will be 5G enabled and just use the cellular network directly.

Even that example is thinking too small. The number of devices using the internet far outstrips the number of humans using the internet. When we hear “internet,” we think of email and Facebook, but we’re just hitchhikers on the internet. Our frail human internet activities are drowned out and lost among the robots using the internet. Medical devices, HVAC systems, militaries, space ships, trucks, fridges, cars, and buildings are the real users of the internet. They’re the real beneficiary of 5G.

The types of speeds 5G brings to the table means that devices can do less processing themselves. Going back to the cell phone as an example, when I play a game on it, my phone has to do all that computational stuff to keep the screen and sound going. That means my phone needs a beefy CPU, a big battery, and good heat dissipation. Contrast that with a 5G network. The same game won’t actually be played on my phone – it will be played on some behemoth gaming computer on the internet and my phone will just tell it what to do. The behemoth will send constantly updating screens showing me the results of my actions as fast as if my phone were doing the processing itself, but my phone is not being dragged down with the heat and battery issues. This shift in processing will change how all devices are made, not just cell phones. They’ll be able to be smaller if that’s desirable, and they’ll be able to operate in more places where current generation devices can’t go because of these limitations.

What’s with the 5G conspiracy nuts?

There’s no medical evidence that 5G causes humans problems any more than wifi does, or 2G, or popsicles. It’s also pretty obvious that 5G doesn’t cause COVID. All that stuff is lies from people with ludicrous agendas or from people with genuine mental health issues, and you don’t need to spend any time thinking about it. A quick read on radio waves, and the visible and invisible spectrum, will serve as a good foundation as to why these people are wrong if you’re interested.

Personally, my next phone will likely be 5G. But, that won’t be the point of the new phone. I won’t upgrade just for the sake of upgrading, but eventually, the only phones I will be able to buy will be 5G capable. While the internet of things will benefit greatly from 5G, I am only a slow meat sack staring blankly at my phone and it’s fast enough for my primitive brain. That’s not to say I won’t enjoy the 5G speeds when I finally get them, it’s just not worth the money for us humans yet.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Everything is more than one thing

One of my favourite TV shows is Halt and Catch Fire. There’s a scene where one of the characters’ daughter behaves like a total idiot at dinner one night by suddenly becoming a vegetarian and refusing to eat dinner thus causing a huge scene between her Dad and the women he has over for dinner, only to come down in the morning to eat meat for breakfast. She explains it away by saying “nothing is one thing” and her Dad, surprisingly, agrees.

Nothing is one thing

I think about that scene a lot because my initial reaction was that “nothing is one thing” is at least as stupid and meaningless as my all-time hated phrase “it is what it is”. But over time I’ve come to realize it is true. We live in a polarized world where we want everyone to be one thing. We want all our friends to be liberals or conservatives, we want everyone to be gay or straight, rich or poor, smart or dumb; anything specific that helps us pigeonhole people in order to make sense of the world. But, in reality, nothing is one thing.

I’m taking that attitude into 2021 with me and it will be reflected a little more in the style and content of this newsletter you’re reading now. I’ve been trying to make this newsletter one thing: a tech newsletter. And while it will definitely remain tech-centric, I’m not going to kill myself trying to produce long-form posts in great detail every single time. I will still do some of that because I like that, but there will be some shorter-form posts from time to time because I like that, too. Hopefully, you’ll like it as well.

World view

I recently posted this on Facebook and on my Fediverse account at Hackers.town, and I wanted to give it a wider audience here.

There's a scientific observation named “world view”. Essentially, it means that in order to understand something, it has to fit into your world view framework. Otherwise it doesn't make sense to you, and you can't understand it.

Being aware of this constraint is one way in which scientists try to avoid bias in their work.

The best example is witchcraft. A tribe consults the oracle on the potential success of an upcoming hunting trip. The oracle throws chicken bones and declares it will be a success. Yet, the hunters come back empty handed. An investigation into what went wrong ensues. Where the chicken bones too old? Were they not cast right? Were they read wrong?

But at no point does anyone question that magic just doesn't exist because that's part of their world view and they can't understand a world without magic.

Somewhere in here is an explanation of what's happening to Trump right now.

Regardless of where you stand on the political spectrum, I think we can all agree that there’s a massive mental meltdown in play at the White House these days. It’s painful to watch it roll out because it’s so obvious that Trump and his ever-decreasing pool of friends have all lost their minds completely. The only explanation I can think of for this crazy behaviour is the world view thing. I can’t relate to them, and they can’t relate to virtually any sane person anymore, because their world view doesn’t include Trump not being President. I’ve never seen anyone so unprepared for something they had four years to prepare for.

Sleeping while driving

Here’s another case of strange behaviour. When automated cars became a reality, I predicted that the highest level of crazy we’d see first was drunk people letting their cars drive them home. But, as usual, I was wrong. It was those damn Canadians sleeping while their car zoomed along at 150 km/h.

The current batch of fledgling automated cars on the roads today are not designed for that. The driver is required to be awake and able to override the car at any time because guess what? The current batch of fledgling automated cars on the roads today tend to do things like drive into other cars and kill pedestrians.

But let’s look at this a little deeper. It’s bad enough the tired Canadians were asleep while their car drove unattended on public roads. But also – it was going 150 km/h. There is literally no place in Canada that has a 150 km/h speed limit. I’ve never seen a speed limit above 120 km/h so a car going 150 is definitely going to attract the attention of any police officer in rage. So, like most crimes, the bad guys were apprehended solely based on their own silliness.

Buh-bye 2020

The “buh-bye” phrase is a holdover from my Army days. Two guys I served with, James and Jav, would say that to each other all the time when they parted. I don’t know why, but it has stuck with me for decades. I don’t actually say it much, but I think it a lot. But in this instance, I am happy as hell to say buh-bye to 2020.

There has been a lot of pain to go around in 2020. Covid took, and continues to take, a big toll on our society. We’ve each learned, with painful precision, exactly how much “scale” plays in our daily lives. The only reason we have things like restaurants and airplanes and public transit and stores is that we have a lot of people using them which drives the cost down and makes these things affordable. When those people go home to isolate, the press of humanity subsides, and businesses close, many forever.

We’ve also learned a lot about “Just In Time” (JIT) inventory which is the holy grail of successful retailers. Big retailers like Walmart strive to eliminate tying up their money in stock sitting in a warehouse doing nothing, and instead try to have new inventory arrive “just in time” to re-stock the shelf when the last thing is purchased. That’s another innovation that keeps costs low, but Covid has taught us that we’re all in this together and when the whole world goes home to isolate, there is no residual inventory sitting anywhere to tide us over, so shelves almost immediately go bare and prices start to creep up.

Personally, 2020 had extra pain for me because I lost my Dad. He died unexpectedly while doing what he loved – puttering around his lakefront property. Intellectually, I know I am at the age when this type of thing happens, and while I am sad, I am OK. But I also feel lonely. Losing someone who has always been in the background fabric of my life is a new experience for me. One less ally in the world.

So, buh-bye 2020 – I won’t miss you at all, but I am looking forward to 2021. Vaccinations have started and while I know we’re a long way away from normal, it’s nice to have some light at the end of the tunnel. It’s nice to have hope.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---

Managing The SSD Life Cycle In A Data Center

Image courtesy of the author.

If you manage or buy server hardware, you will be buying Solid State Drives (SSD) these days. Spinning platters are a thing of the past; they are good enough for personal computers, but unable to withstand the harsh data center environment. But before you spend $50,000 on SSD drives, you need to know what to expect. Or, in other words, how long should your drives last?

Site Reliability Engineers need to answer these types of questions. There are many people inside an organization who may need to know this ranging from hardware teams that know every nuance of SSD drives, all the way over to the finance department who just needs to know how you came up with this dollar amount for drive replacement this year.

The lifespan of an SSD drive can’t be measured solely in time because the workload of the drive plays a big part. A drive that is used for booting a server will last a very long time because it has minimal read and writes occurring. On the other hand, a drive that is used for a CDN cache will have a much shorter life span because it is constantly being read from and written to. It is for this reason that SSD drive endurance and lifespan is expressed in how much data can be written to it.

The reason SSD drives are so much faster than spinning platter drives is because SSD drives use technology more similar to RAM than moving disks. When a hard drive is asked to retrieve data, it has to do a lot of mechanical movement to find it. An SSD drive doesn’t move at all. It just looks up the data from its static memory chips which is a much faster process.

To determine how much use you will get out of an SSD drive you need to know three basic things:

1. How big is the drive?

2. How long is the warranty?

3. What is the manufacturer’s endurance rating?

Manufacturer’s endurance rating terminology

The manufacturer’s endurance rating will be expressed in one of three ways:

Drive Writes Per Day (DWPD):

This is a rating of how many times the drive could be entirely written over every single day of its life before the manufacturer will no longer guarantee its performance. Which is just another way to say “warranty period”. If the entire drive is written over less than this value every day, then the drive will work for its stated warranty period.

Drive endurance is generally no longer expressed using DWPD, but I find it helpful to reduce the TBW/PBW numbers back to DWPD numbers (more on those two methods below).

Terabytes Written (TBW)

This is a rating of how many TBs of data can be written to the drive over its entire life within its warranty period and still perform as expected.

Note that TBW can be expressed in Terabytes or Petabytes and is usually annotated like TBW(TB) or TBW(PB) to indicate form which is being used.

Petabytes Written

This is the same as TBW, but expressed in Petabytes to accommodate today’s larger drives.

The TBW and PBW are more useful metrics because we’re dealing with just a single metric — how much data, period. The DWPD leaves open questions because of how it presents writes based on days.

Warranty period

I think we all understand the concept of a warranty period, but it’s worthwhile quickly touching on it in this context.

We should never plan for our hardware to remain in service past its warranty date. — Every SRE Ever

Warranty is the time frame the manufacturer thinks the product will function acceptably. In personal life, I think we all like stuff that outlives its warranty. But, we don’t want that in the data center. We should never plan for our hardware to continue past the warranty date because equipment becomes unpredictable at that point. Relying on an out of warranty drive could cause unexpected outages or, even worse, perhaps damage to some other part of the server. Always forecast your drive replacements based on the warranty period.

Drive capacity

Now that we understand that the amount of data written and the warranty period, it becomes more obvious how drive capacity factors into the lifespan calculation.

Consider two drives with the same warranty period (5 years) and the same endurance rating (8760 TBW), but different capacities. A 960GB drive will have a DWPD rating of 5. A 480GB drive will have a DWPD rating of 10.

Does that mean the 480GB drive has more endurance?

Technically, the drives have the same endurance because although the 960 has a smaller DWPD rating, it is writing to twice the space as the 480GB drive. Half the DWPD over twice the space is pretty much the same thing. So the answer lies in how you will use that drive. If it will be pressed into service in a high read/write situation, then the smaller drive would be a better choice. The downside is, of course, it’s a smaller drive.

Let’s play with the warranty period a bit. The standard warranty in the industry is 5 years, but what if it was a 7 year warranty? The 960GB drive then drops to 3.5 DWPD and the 480GB drive drops to 7.1 DWPD. That makes sense because we’re now spreading that 8670 TBW over a longer period, therefore less can be written per day.

Lest you think I am a mathematical savant, I will provide you with this link which makes these conversions easier:

https://wintelguy.com/dwpd-tbw-gbday-calc.pl

Calculate how long it will last

Now that we understand all the pieces, let’s get down to figuring out how long your drive will last. The best predictor of the future is the past, so hopefully you have some drives in service now that you can look at.

Auditing current drives

The two things you will want to understand about your current drives are:

1. What shape are they in now, and

2. When did they go into service?

What shape are the drives in now?

Smartmon Tools can help here and every Linux distribution I’ve come across has the tools installed or available in its repositories. There is a lot of criticism about how Smart Tools expresses drive wear. Some of it is valid, but in my experience it works “Good Enough”. Nothing can tell you with 100% certainty how long a drive will last. There are too many variables ranging from heat in the chassis to manufacturing tolerances. The best we can do is collect data and try to make supportable forecasts from it.

SSD drive wear is shown via the Media Wear Indicator in the Smart Tools output. I’ve had to truncate the output a bit because Medium’s formatting doesn’t show all the columns nicely.

/smartctl -A /dev/sdc | egrep 'MediaWearoutIndicator' ID# ATTRIBUTENAME VALUE WORST TYPE UPDATED MediaWearoutIndicator 0 95 095 Oldage Always

From this we can see that the /dev/sdc drive is in good shape at 95/100 wear. That’s good info, but we really don’t know how to evaluate it unless we also know how old the drive is.

How old is the drive?

Hopefully, you have some historical paperwork to tell you when your drives went into use. But, if not, you can make some educated guesses. I have several drives with no history, so I looked at the earliest file on them. These are Linux machines and I know that when we format the drives for use, a folder named lost+found is created. Because there’s no obvious reason for anyone to touch that folder, it is reasonable to assume that the date of that folder reflects the date the drive went into service.

You can build a myriad of scripts to determine the oldest file on a disk. Here’s one that checks the /, /boot, /var/log, and /var/cache partitions. I redirect the output to a file which ends up in CSV format because of the comma formatted output. It’s not elegant, but it need not be. Feel free to modify it for your own use:

echo -n hostnameecho -n “,smart_boot,“cd /boot/date -r ls -1t | sed "s/://" | tail -n1 “+%Y”

echo -n hostnameecho -n “,smart_cache,“cd /var/cache/date -r ls -1t | sed "s/://" | tail -n1 “+%Y”

echo -n hostnameecho -n “,smart_log,“cd /var/log/date -r ls -1t | sed "s/://" | tail -n1 “+%Y”

echo -n hostnameecho -n “,smart_slash,“cd /date -r ls -1t | sed "s/://" | tail -n1 “+%Y”

It gives output like this:

server1,smartboot,2016 server1,smartcache,2014 server1,smartlog,2014 server1,smartslash,2015

Assuming each of these partitions are on a different drive, we can see that we have some drives that are out of, or just about to fall out of the typical 5-year warranty window.

Put it all together

Now we have a good idea of our drive life. Using good old /dev/sdc — which is mounted on /var/log — I can see that even though the drive has lots of wear left, it is outside of the warranty period. Therefore, I am at risk of failures regardless of the drive wear left, so it should go on my replacement list.

With this data we can see that the drive model I’m using on /dev/sdc is obviously overkill. It should not have 95% wear left after 6 years. This is an opportunity to review the drive specs and perhaps save some money replacing it with a drive with less endurance. What I am looking for is a drive about 4 years old with about 20% wear left. That is the sweet spot of efficiency. It is a drive that will likely last the warranty period, but not too long after. It is a drive that I am spending the “right” amount of money on. Even if I don’t encounter a drive like that in my inventory, I can at least look at the specs and wear of the drives I do have to give me some upper and lower limits to work within to help source a more suitable drive.

These tools and concepts should give you enough basis to determine the drive replacement cadence you will need to establish. I use variations of these script and tools to build CSV files that I can share with almost everyone in the company, regardless of their technical expertise. I can show the hardware teams how I determined that the current drives we’re buying are (or are not) sufficient for our needs. I can also show the less technical procurement people how I determined that I need $X budget for drive replacement this year.

my shorter content on the fediverse: https://the.mayhem.academy/@jdw

---