Today I Learned... (warning, knowledge here is 'raw', so it may not be completely accurate, but represents my understanding of the tech at the time. No warranty is expressed or implied.)

til.md

Fog Computing

In a traditional IoT architeture, you have your normal "device" that transmit messages to the cloud and receive a response. However, this approach is "slow" (high latency) - the device must wait for the following steps to occur:

1. the data being transmitted across the Internet to the cloud provider

2. the cloud provider doing its own processing of the data

3. the cloud provider transmitting the data back to the device

This high latency may be undesirable if we want the device to quickly respond to an event rather than simply "wait". For example, if the device is a security camera, we want to be able to identify intruders immediately, not after a few seconds of cloud processing (enough time for the intruders to get away). Similarly, a medical device may need to provide effective treatment to a patient in a life-or-death situation immediately, not after a few seconds (when the patient is already dead).

In addition, the transmission of data across the Internet could lead to increased bandwidth and energy consumption, which can increase the cost of running the device.

We could reduce the latency of IoT requests by creating a "fog layer" (that consists of several 'fog nodes') that sits in-between the cloud and the device. The device, instead of sending all requests to the cloud, can decide whether to send the data to the "fog layer" or to the cloud.

• If the device needs to receive an immediate response, it sends the data to the nearest fog node. The fog node then processes the data and sends the response back. The fog node then sends the data to the cloud for storage.
• If the device needs to process the data quickly (but not immediately), it can send the data to a fog node. The fog node can then send it off to a centralized "aggregate fog node", who will then process the data and send the response back to the device. The "aggregate fog node" can then send the data off to the cloud for storage.
• If this isn't really time-sensitive data, then the device can send the data to the cloud as normal.

Not only are we able to reduce the latency whenever we need to, but we also reduce the bandwidth and energy consumption of the IoT device.

Here's the catch. You need to actually build the fog layer first. The energy consumption of these nodes are much higher than that of the traditional cloud, and you need to manage all these nodes as well (you don't want your lovely fog computing system to blow up the second a node disconnects). There's a lot of challenges involved in setting up this architecture. However, if you can get it up and running, it may be incredibly useful for a large enterprise.

An interesting side-note is that "fog nodes" are mobile (and so can be placed in areas that are 'remote' and difficult for traditional cloud infrastructure to reach) and can easily join or leave the "fog layer" whenever necessary. So "fog nodes" themselves appear to be an incredibly 'agile' technology.

Monoliths and Microservices, Using "Star Wars" As An Analogy

The Monolithic Republic has tradition by its side. It has always existed in some form for milliennia, and so people are naturally inclined to it. It is a very centralized entity where everything is handled by the Galactic Senate (technical translation: the application). Any request you make goes straight to the Galactic Senate to be processed.

Pros of the Monolithic Republic:

1. It's easy to get the "big picture", simply by studying the Galactic Senate in detail. If you understand the Galactic Senate, then you understand the "big picture".
2. It's relatively speedy to make a "request". You just send a request to the Senate, and the Senate responds back. (Technical translation: Network latency is reduced, due to only a single network call being made.)
3. "Security" is much easier to provide here. It's harder for outsiders to subvert or control the Galactic Senate, due to the Senate's centralized nature (only one entity in charge).
4. Because the Monolithic Republic has a very rich tradition, there's a lot of literature about how to "change" it (technical translation - develop new features for the application and then deploying it quickly so that people can use it). Due to the presence of this literature, people are used to dealing with the Monolithic Republic, so the process of working with the Republic is relatively easy.

Cons of the Monolithic Republic:

1. Sloth. The Galactic Senate may be huge, because the Senate must constantly change and adapt to meet the needs of the people. Thus, it may be difficult for someone to fully study the "big picture" - much less even "change" it. It's also hard to roll out "changes" to the galaxy (technical translation - every time you need to make a minor modification to the application, you need to re-deploy the whole application...the time it takes to re-deploy the whole application increases as the application itself increase).
2. Corruption. Suppose the Galactic Senate receives more requests than expected - 120 requests instead of the expected load of 100 requests. How do we scale? We could simply spin up another instance of the Galactic Senate, so you have two Galactic Senates running at the same time handling the 120 requests. That will work. But this is incredibly inefficent - we're wasting a lot of resources. After all, your two Senates can handle 200 requests in total, but you're really processing only 120 requests. Not just that, but we also don't take into account where those requests are going - if all the additional 20 requests are going to the Hyperspace Regulation Committee, then we are needlessly duplicating all the other committees that the Senate provides. Resources are being wasted on entities that have no reason to even exist.
3. Decadence. The fundamental nature of the Senate cannot be changed once it has been created. Society has ossified, and you're stuck with the decisions that was already made years ago...unless you're willing to invest the time in destroying the system and rewriting it all from scratch (technical translation: If we write the application in one programming language or technology stack, we are stuck with that language/stack, and can't switch to a different language/stack that might be better suited for the task...unless we do an expensive rewrite).

As a result of the Republic's "sloth", "corruption" and "decadence", the Republic would eventually be overthrown by demagogues promising a Microservices Empire. The Empire avoids the failed policies of centralization inherent within the 'Old Republic'. Instead, each committee act as a mini-Senate in charge of a specific domain. If the people want something, they will simply contact the committee in charge, and the committee responds. The committees can also easily communicate with each other (technical translation - they do so through an open protocol like HTTP).

Pros of the Microservices Empire:

1. Easier to change and modify. It's easy to understand what a single committee is doing (since each committee is an expert at one, and only one domain). It's easy to replace one outdated committee with another, more modern committee written in a different language/stack. It's easy to add new features and to re-deploy a single committee, without ever having to touch all the other committees (though there is a learning curve involved in learning how to package and deploy the committees separately from one-another). The 'Empire' may be far more responsive to the changing needs of galactic society than the Old Republic ever was.
2. Scaling is much more efficent now. To handle additional load to the Hyperspace Regulation Committee...you only need to spin up another instance of the Hyperspace Regulation Committee. There's no "waste" involved in creating unnecessary committees.
3. There's also a lot of literature about microservices as well (though they were generally referred to as SOA: Service-Oriented Architecture). In addition, more literature is being developed every day, with more "patterns" and tools being built to handle the various challenges within the Microservices Empire. So it's not as if the Imperial reformers are stuck wandering blindly.

Cons of the Microservices Empire:

1. Complexity Shifts to the Infrastructure - While it is now easier to understand a single committee, it's much harder to understand the interactions between the committees (as well as interactions between the commitees and the 'outside world'). Managing all the committees become far more difficult and confusing, as it requires a lot of monitoring and control over the flow of communication between the different committees. And you have to do so lest disaster may hit - for example, the standard 'Cascading Failure' disaster ("Committee A goes down, causing Committee B to go down, causing Committee C to go down...etc."). Always a hit at parties.

• That being said, automation may help you deal with this challenge (it's easy to minimize the possibility of a Cascading Failure by using 'Expotential Backoff'). But even automation has to be maintained.

2. Monitoring/Managing Headache - Instead of simply monitoring and managing a centralized Senate, you must now monitor and manage every single committee to find out any problems that might arise. How are you going to read all those log files? What about understanding the log files?
3. Vulnerability - There's a much larger "attack area" for enemies to exploit. In the days of the Old Republic, if you want to cause mayhem, you have to need to subvert the whole Galactic Senate. But now, you could make mischief simply by seeking out a insecure committee.
4. Reform Is Hard - Conversion of an existing Monolithic system to the "Microservices" system can be very difficult and time-consuming (as the original Imperial pioneers learned the hard way). It's usually much easier to build a Microservices Empire from scratch...but you may lack this luxury if you're inheriting an existing "legacy" system that people are dependent on.

• But see Martin Fowler's blog post "Monolith First", where he argues that most galaxies may not need the complexity of managing a Microservices Empire (so jumping straight to the Empire phase may be very foolhardy). Instead, galaxies should build a Monolithic Republic first and gradually reform it to a Microservices Empire if necessary.

5. Requests May Take Longer - That's because we may be making multiple requests to different committees, instead of making one single request to the Galactic Senate like we did during the Old Republic.
6. The cutting edge is merely an euphemism for the the Bleeding Edge. With microservices still being fairly new, people may encounter more challenges and diffculties that we have not yet accounted for or fully appreciated. It may take years for the results of an architecture to show itself.

Of course, nothing can last forever. The Old Republic may have fallen, but the Microservices Empire still has to prove itself. Even now, I hear reports of the "rebel scum" that are quietly resisting the Microservices Empire. And the galaxy is prone to fads and trends under the guise of "progress", meaning that new approaches to organizing complex systems may arise. Keep your eye on the present, but remember to alway look ahead to the future.

Related Reading: "Microservices: a definition of this new architectural term", by James Lewis and Martin Fowler, "Mircoservices Tradeoffs", by Martin Fowler

"Monolithic Core", Using "Star Wars As An Analogy"

Though the Microservices Empire may have its merits, its implementation may have left something to be desired. Its degeneration into an unmaintainable mess may have been preventable...but that's really a debate for the academics. The truth is that it failed, and it failed hard. And so the "rebel scum" overthrew their Imperial overlords, sending them off into the outskirts of the galaxy.

The rebels naturally set up a New Monolithic Republic. But though the New Monolithic Republic may have its merits, its implementation left something to be desired, and the Imperial ex-overlords are able to gain support amongst the galactic citizerny (under the "Imperial Stormtrooper With a Human Face" propaganda campaign). The truth is that there are trade-offs involved in whatever form of government you choose for your galaxy. There is no such thing as a 'free lunch'.

But centralized monoliths and decentralized microservices are merely two extremes on a spectrum of possible '-alizations'. Most galactic governments will be within that spectrum, and won't fall prey to extremes. Could we therefore pick a "happy medium" that minimizes the negative consequences of our choices (at the cost of minimizing the positive consequences of our choices)?

After years of conflict and disagreement, the feud between New Monolithic Republic and the Microservices Empire finally ended as both sides unify together into the "Galactic Federation". Despite the 'federated' name, most power lies within the Monolithic Republic. It holds most of the resources, and call most of the shots. The Federation can be said to have a "Monolithic Core".

However, the Microservices Empire still exist on the outskirts of the galaxy. If the Core has certain 'dirty work' that needs to be done (but is not central to the 'core' features of the Core), it outsources off to the Empire (technical translation: "background jobs, notifications, or other small subsystems"). The Monolithic Core wants to retain its control over the Federation, so it has full possession on the 'data' that underlines the galaxy while the Empire processes any data that the Core sends off (technical translation: we want stateless microservices here, please don't give each microservice their own database in this model).

Some people may confuse the Federation as being another type of "Microservices Empire". It's not. In a 'pure' Micorservices Empire, there is no "core" to speak of - every committee acts independently and can communicate with other committees whenever they need to. This makes the Microservices Empire way more flexible (and way more complex as well). In the Federation approach though, there is clearly one entity running the show (the Monolithic Core). This reduces the technical complexity of the system to a more managable level, but also reduces the flexibility.

Some Imperial reformers advocate a Federation as a temporary stage towards a "pure" Microservices Empire. In this approach, the Monolithic Core would slowly decay away as more and more committees split off and join the Microservices Empire. I'm not sure whether I would endorse such an approach though - if the Federation works perfectly, why exactly do we need to switch?

Related Reading: Monolithic Core versus Fully Microservices Architecture, by Florian Motlik

Debugging Thought #1

Assume you have written this code:

def complex_code( input )
    a = AlphaProxy.call( input ) # returns either true or false
    b = BetaProxy.call( input ) # returns either true or false
    c = GammaProxy.call( input ) # returns either true or false

   if (a && b && c )
       d = true
   else
       d = false
   end
   
   if ( d )
      e = DeltaProxy.call( input ) # returns either true or false
   end
   
   if ( d == false || e == false )
      raise "Horrible Exception"
   else
      return true # complex code was successfully executed
   end
end

Or, in pseudocode:

Use the input provided to make three proxy calls to Alpha, Beta, and Gamma (original proxy calls).

If original proxy calls are successful, use the input provided make a proxy call to Delta.

If all proxy calls are successful, return true.

If the original proxy calls fail or the Delta proxy call fails, raise an exception. The exception message will not distingush between a failure in the original proxy call or the Delta proxy call.

So there are essentially four different proxy calls you need to make, and if any of them break, the whole system breaks.

Assuming you can't change/refactor the code to better highlight where the error took place, what proxies should you check first? There's two options:

1. Check if the Delta Proxy Call is working, since everything is dependent on it. If it is working, then the problem is with the original proxy calls.
2. Check if the original proxy calls are working, since you need them to work before you can make the Delta Proxy call. If the original proxies are working, then the problem lies with with the Delta proxy call.

The worst case-scenario is that you must check 4 different proxies, just to debug one method. It's time-consuming, but it's possible.

Alternative Methods Of Building "URL Applications"

I first learned of URL Applications when I was introduced to https://itty.bitty.site/ (though mermaidjs also used URL fragments as well). However, this Hacker News post revealed two other approaches to build URL applications.

1. Hashify.me, which was built in 2011. Has a much prettier Markdown editor than itty.bitty.site honestly, but the generated URL is allegedly longer than the content (to compenstate, a bit.ly URL shortner is included with the Markdown editor for you to use when you want to). De facto limit on how much content can be used - will raise a "Request-URI Too Large" error. Not sure how they're generating the URL as well, at least not at first glance.

2. Data URI schemes, as detailed on Wikipedia.

Examples:

data:text/plain;base64,SGVsbG8sIHdvcmxkISBJJ20gZG93bmxvYWRlZCB2aWEgImRhdGE6dGV4dC9wbGFpbjsuLi4iIFVSTCB1c2luZyA8YSBkb3dubG9hZD0iZmlsZV9uYW1lIi4uLj4uDQpNeSBiaXJ0aHBsYWNlOiBodHRwOi8vc3RhY2tvdmVyZmxvdy5jb20vcXVlc3Rpb25zLzY0Njg1MTcvDQoNCk1vcmUgYWJvdXQ6DQpodHRwOi8vd3d3LnczLm9yZy9UUi9odG1sL2xpbmtzLmh0bWwjYXR0ci1oeXBlcmxpbmstZG93bmxvYWQNCmh0dHA6Ly93d3cudzMub3JnL1RSL2h0bWwvbGlua3MuaHRtbCNkb3dubG9hZGluZy1yZXNvdXJjZXMNCg0KQnJvd3NlciBzdXBwb3J0OiBodHRwOi8vY2FuaXVzZS5jb20vZG93bmxvYWQ= 

 data:text/html,<title contenteditable>Hello World</title> <style contenteditable>head, title, style {display: block;}</style> 

  data:text/html,<html contenteditable></html> 

Bash Script to generate data URIs:

  #!/bin/sh -eu
  printf 'data:%s;base64,%s' "$(file -bi "$1"|tr -d ' ')" "$(base64 -w 0 "$1")"

See Also: How to make a real-time in-browser editor with the HTML5′s contenteditable attribute