Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Things I believe

Things I believe

This is a collection of the things I believe about software development. I have worked for years building backend and data processing systems, so read the below within that context.

Agree? Disagree? Feel free to let me know at @JanStette. See also my blog at www.janvsmachine.net.

Fundamentals

Keep it simple, stupid. You ain't gonna need it.

You should think about what to do before you do it.

You should try to talk about what you’re planning to do before you do it.

You should think about what you did after you did it.

Be prepared to throw away something you’ve done in order to do something different.

Always look for better ways of doing things.

“Good enough” isn’t good enough.

Code

Code is a liability, not an asset. Aim to have as little of it as possible.

Build programs out of pure functions. This saves you from spending your brain power on tracking side effects, mutated state and actions at a distance.

Use a programming language with a rich type system that lets you describe the parts of your code and checks your program at compile time.

The expressivity of a programming language matters hugely. It’s not just a convenience to save keypresses, it directly influences the way in which you write code.

Choose a programming language that has a good module system, and use it. Be explicit about the public interface of a module, and ensure its interals don't leak out to client code.

Code is a living construct that is never “done”. You need to tend it like a garden, always improving and tidying it, or it withers and dies.

Have the same high standards for all the code you write, from little scripts to the inner loop of your critical system.

Write code that is exception safe and resource safe, always, even in contexts where you think it won’t matter. The code you wrote in a little ad-hoc script will inevitably find its way into more critical or long-running code.

Use the same language for the little tools and scripts in your system too. There are few good reasons to drop down into bash or Python scripts, and some considerable disadvantages.

In code, even the smallest details matter. This includes whitespace and layout!

Design

Model your domain using types.

Model your domain first, using data types and function signatures, pick implementation technologies and physical architecture later.

Modelling - the act of creating models of the world - is a crucial skill, and one that’s been undervalued in recent years.

Implement functionality in vertical slices that span your whole system, and iterate to grow the system.

Resist the temptation to use your main domain types to describe interfaces or messages exchanged by your system. Use separate types for these, even if it entails some duplication, as these types will evolve differently over time.

Prefer immutability always. This applies to data storage as well as in-memory data structures.

When building programs that perform actions, model the actions as data, then write an interpreter that performs them. This makes your code much easier to test, monitor, debug, and refactor.

Dependency management is crucial, so do it from day one. The payoff for this mostly comes when your system is bigger, but it’s not expensive to do from the beginning and it saves massive problems later.

Avoid circular dependencies, always.

Quality

I don’t care if you write the tests first, last, or in the middle, but all code must have good tests.

Tests should be performed at different levels of the system. Don’t get hung up on what these different levels of tests are called.

Absolutely all tests should be automated.

Test code should be written and maintained as carefully as production code.

Developers should write the tests.

Run tests on the production system too, to check it’s doing the right thing.

Designing systems

A better system is often a smaller, simpler system.

To design healthy systems, divide and conquer. Split the problem into smaller parts.

Divide and conquer works recursively: divide the system into a hierarchy of simpler sub-systems and components.

Corollary: When designing a system, there are more choices than a monolith vs. a thousand “microservices”.

The interface between parts is crucial. Aim for interfaces that are as small and simple as possible.

Data dependencies are insidious. Take particular care to manage the coupling introduced by such dependencies.

Plan to evolve data definitions over time, as they will inevitably change.

Asynchronous interfaces can be useful to remove temporal coupling between parts.

Every inter-process boundary incurs a great cost, losing type safety, an making it much harder to reason about failures. Only introduce such boundaries where absolutely necessary and where the benefits outweigh the cost.

Being able to tell what your system is doing is crucial, so make sure it’s observable.

Telling what your system has done in the past is even more crucial, so make sure it’s auditable.

A modern programming language is the most expressive tool we have for describing all aspects of a system.

This means: write configuration as code, unless it absolutely, definitely has to change at runtime.

Also, write the specification of the system as executable code.

And, use code to describe the infrastructure of your system, in the same language as the rest of the code. Write code that interprets the description of your system to provision actual physical infrastructure.

At the risk of repeating myself: everything is code.

Corollary: if you’re writing JSON or YAML by hand, you’re doing it wrong. These are formats for the machines, not for humans to produce and consume. (Don’t despair though: most people do this, I do too, so you’re not alone! Let's just try to aim for something better).

The physical manifestation of your system (e.g. choices of storage, messaging, RPC technology, packaging and scheduling etc) should usually be an implementation detail, not the main aspect of the system that the rest is built around.

It should be easy to change the underlying technologies (e.g. for data storage, messaging, execution environment) used by a component in your system, this should not affect large parts of your code base.

You should have at least two physical manifestations of your system: a fully integrated in-memory one for testing, and the real physical deployment. They should be functionally equivalent.

You should be able to run a local version of your system on a developer’s computer with a single command. With the capacity of modern computers, there is absolutely no rational reason why this isn’t feasible, even for big, complex systems.

There is a running theme here: separate the description of what a system does from how it does it. This is probably the single most important consideration when creating a system.

Building systems

For a new system, get a walking skeleton deployed to production as soon as possible.

Your master branch should always be deployable to production.

Use feature branches if you like. Modern version control tools make merging easy enough that it’s not a problem to let these be long-lived in some cases.

Ideally, deploy automatically to production on every update to master. If that’s not feasible, it should be a one-click action to perform the deployment.

Maintain a separate environment for situations when you find it useful to test code separately from production. Avoid more than one such extra environment, as this introduces overheads and cost.

Prefer feature flags and similar mechanisms to control what's enabled in production over separate test/staging environments and manual promotion of releases.

Get in the habit of deploying from master to production from the very beginning of a project. Doing this shapes both your system and how you work with it for the better.

In fact, follow all these practices from the very beginning of a new system. Retrofitting them later is much, much harder.

Technology

Beware of hyped or fashionable technologies. The fundamentals of computer science and engineering don’t change much over time.

Keep up with latest developments in technology to see how they can help you, but be realistic about what they can do.

Choose your data storage backend according to the shape of data, types of queries needed, patterns of writes vs. reads, performance requirements, and more. Every use case is different.

That said, PostgreSQL should be your default and you should only pick something else if you have a good reason.

Teams

Hiring is the most critical thing you do in a team, so allocate time and effort accordingly.

Use the smallest team possible, but no smaller.

Do everything you can to keep the team size small. Remove distractions for the team, delegate work that isn’t crucial, provide tools that help their productivity. Increasing the team size should be the last resort.

Teams need to be carefully grown, not quickly put together. When companies brag about how many engineers they’re planning to hire in the near future, this is a massive red flag.

New systems are best designed by a small numbers of minds, not committees. Once the structure of the system is clear and the main decisions made, more people can usefully get involved.

Code ownership may be undesirable, but it’s important to have someone who owns the overall vision and direction for a system. Without it, the architecture will degrade over time as it gets pulled in different directions.

Prefer paying more to get a smaller number of great people instead of hiring more people at a lower salary.

Use contractors to bring in expertise, to ramp up a project quickly, for work on trial projects, or to deal with specialised work of a temporary nature. Don’t use contractors as a substitute for ordinary staff. (I say this as a contractor.)

If you’re hiring contractors because you can’t get permanent staff, you’re not paying your permanent staff enough.

As a team lead, if things go well, give your team the credit. If things go badly, take the blame yourself.

A team member who isn’t a good fit for the team can kill the performance of the whole team.

Inter-personal conflict is likewise poison to a team. While it may be possible to resolve underlying issues, if it proves difficult, it’s usually better for everyone to accept this and move people on.

Many people do “agile” badly, but that doesn’t mean that “agile” is bad.

Out of all the agile practices commonly used, estimating task sizes and trying to measure project velocity is the least useful. (Its utility is often less than zero).

People

Be kind towards others, everyone faces their own battles no matter how they appear externally.

Be kind to yourself. What you’re doing is hard, so accept that it sometimes takes longer than you’d like and things don’t always work out.

Don't let your own desire to get things done quickly turn into undue pressure on your peers.

The more certain people are about their opinions, the more you should take them with a pinch of salt.

Imposter syndrome is a thing, as is the Dunning-Kruger effect. So judge people on their contributions, not on how confident they seem.

Personal development

Always keep learning.

Read papers (of the scientific variety).

Read the classics. So many “new” ideas in software development have actually been around for decades.

Rest. Don’t work more than 8 hours a day. Get enough sleep. Have interests outside your work.

Work efficiently instead of very long days. Try to remove distractions as much as possible, for example pointless meetings. (As an aside, some meetings are actually useful.)

When looking for a solution to a problem, don’t just pick the first one that comes to mind. Learn to look for alternatives and evaluate them to find the best choice.

Don’t work in isolation. Share ideas, talk new ideas and designs over with your peers.

Don't be afraid to have strong opinions, but listen carefully to the opinions of others too.

@JonNorman

This comment has been minimized.

Copy link

JonNorman commented Jan 26, 2020

You should have at least two physical manifestations of your system: a fully integrated in-memory one for testing, and the real physical deployment. They should be functionally equivalent.

A great list, and I agree with almost everything almost entirely. This point is the one I'm least sure about however. With the prevalence of containers and the ease with which one can often have a local instance of the infrastructure running without too much difficulty, what is the benefit of having an InMemoryDataStore vs using your ActualDbDataStore with a local instance for testing and a PROD connection for PROD?

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Jan 26, 2020

This point is the one I'm least sure about however. With the prevalence of containers and the ease with which one can often have a local instance of the infrastructure running without too much difficulty, what is the benefit of having an InMemoryDataStore vs using your ActualDbDataStore with a local instance for testing and a PROD connection for PROD?

Good question. The most important thing is indeed that you can run the system locally. That said, I do think it's valuable to have a reference implementation that you can use to verify the correctness of the one that talks to an actual database. It also ensures that you're developing at the right level of abstraction, not just coding against a concrete implementation. It's a fairly subtle point to be fair, maybe one to expand on somewhere else...

Also: gists can have comments?? :D

@JonNorman

This comment has been minimized.

Copy link

JonNorman commented Jan 27, 2020

It also ensures that you're developing at the right level of abstraction

I like this reason; it joins back to your other points about being scrupulous with interface boundaries and coupling 👍.

Also: gists can have comments?

Apparently! I would have responded via @JanStette as you requested (I didn't want to clog up the gist with comments) but the character limit would've got in th

@aloisdg

This comment has been minimized.

Copy link

aloisdg commented Jan 28, 2020

Code is a liability, not an asset. Aim to have as little of it as possible.

Counter example: APL, Japt, Pyth, etc. CodeGolf is great, but readability is key here. Meaningful or verbose code can help a lot. Clever code or tricks should be use with moderation. Dont remove to much. It is fine to be a bit expressive.

@davorpa

This comment has been minimized.

Copy link

davorpa commented Jan 29, 2020

A great list, men!!!

@Doug-AWS

This comment has been minimized.

Copy link

Doug-AWS commented Jan 29, 2020

I love Brian Kernighan's advice on avoiding clever code:
“Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”

@daarond

This comment has been minimized.

Copy link

daarond commented Jan 29, 2020

On personal development:
Your body can be more than just a meat sack to move your brain around. Don't let sedentary coding kill you, exercise every day.

@flakey-bit

This comment has been minimized.

Copy link

flakey-bit commented Jan 29, 2020

Code is a liability, not an asset. Aim to have as little of it as possible.

Counter example: APL, Japt, Pyth, etc. CodeGolf is great, but readability is key here. Meaningful or verbose code can help a lot. Clever code or tricks should be use with moderation. Dont remove to much. It is fine to be a bit expressive.

I suspect the point is rather "If there is a popular & battle-tested program/library that does what you want, use that instead of writing something new"

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Jan 29, 2020

On personal development:
Your body can be more than just a meat sack to move your brain around. Don't let sedentary coding kill you, exercise every day.

I love this! It reminds me I've been meaning to write a blog post called "running for geeks", I really should get around to it...

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Jan 29, 2020

Apparently! I would have responded via @JanStette as you requested (I didn't want to clog up the gist with comments) but the character limit would've got in th

😆

Also, gists have comments but no reaction emojis on the comments, what's up with that?!

@veeral-patel

This comment has been minimized.

Copy link

veeral-patel commented Jan 30, 2020

Use a programming language with a rich type system that lets you describe the parts of your code and checks your program at compile time.

The expressivity of a programming language matters hugely. It’s not just a convenience to save keypresses, it directly influences the way in which you write code.

Choose a programming language that has a good module system, and use it. Be explicit about the public interface of a module, and ensure its interals don't leak out to client code.

@stettix just curious - what languages do you primarily work in?

@veeral-patel

This comment has been minimized.

Copy link

veeral-patel commented Jan 30, 2020

Build programs out of pure functions. This saves you from spending your brain power on tracking side effects, mutated state and actions at a distance.

@stettix I like this! But how does it work in practice? If you're building a typical REST API, then you probably have controllers which call your business logic, which in turn makes some database calls.

Where would the pure code in this program go? Or is a REST API a bad place to apply this recommendation?

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Jan 30, 2020

@stettix I like this! But how does it work in practice? If you're building a typical REST API, then you probably have controllers which call your business logic, which in turn makes some database calls.

Where would the pure code in this program go? Or is a REST API a bad place to apply this recommendation?

That's a good question, web APIs and data storage often dominate what we think about when building backend services, and we often think about these as "layers" that we structure applications around. There are other ways though - I'd suggest reading about the onion Architecture, hexagonal architecture, or clean architecture for example. Common to all these approaches is that the core of the applicaiton is the pure domain data types and business logic, and both data storage and APIs are things that plug into this core from the outside.

Another way to look at it: what would your application look like if you didn't have web APIs and databases, and it was just implemented in terms of in-memory data structures, and perhaps a simple interface (programmatic or command line for example)? This can be a great way to start work on an application, it frees you up to think about the key aspects of it, and you can evolve and refactor the code quickly, then introduce persistence and APIs later. Even for an established application this can be a useful exercise, to help refactoring the application into a better form.

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Jan 30, 2020

@stettix just curious - what languages do you primarily work in?

I mostly work in Scala, but I've also dabbled in OCaml and Haskell, and I like the look of F# and Reason for example.

@veeral-patel

This comment has been minimized.

Copy link

veeral-patel commented Jan 31, 2020

When building programs that perform actions, model the actions as data, then write an interpreter that performs them. This makes your code much easier to test, monitor, debug, and refactor.

@stettix this sounds really interesting but could you elaborate? I didn't get what you meant by writing an interpreter in particular. I don't think you meant writing a real interpreter, like programming languages have :)

@eatkins

This comment has been minimized.

Copy link

eatkins commented Jan 31, 2020

Choose a programming language that has a good module system, and use it. Be explicit about the public interface of a module, and ensure its interals don't leak out to client code.

You make a lot of great points but I feel like this is easily one of the most under-appreciated aspects of writing maintainable code. When thinking about the public api, you are forced to think clearly about how the module is intended to be used by callers. Invariably you will write code that has downstream consumers. If you made avoidable mistakes in the api design (usually by bloating the api with implementation details), then you create extra pain either for yourself, those downstream or both when you need to evolve the api in the future. Since you will be unable to anticipate all use cases, I advocate for proceeding with a minimal initial public interface, not just explicit, even if a complex internal interface is used to implement it (corollary: one should almost never call internal interfaces in tests -- in scala, tests are ideally implemented in a separate test package to enforce this). This will make api migrations, which are really not that different from database schema migrations, much simpler. The discipline applied here reinforces many of the other good habits that you enumerate.

I also appreciated that you titled this "Things I believe" rather than, say, "Things I know" about programming. So many of the design arguments that I've seen masquerade as fact based when they are really based in belief. Even if you believe you are right, you have to understand what the other person believes and how that is coloring their interpretation of the facts in order to engage them. In my experience, a lot of people are unwilling to admit or unaware of how much of what they do is grounded in belief rather than some mythical objective truth. This is a massive collective blind spot that contributes to a lot of avoidable conflict as well as bad outcomes since, when arguing over beliefs, usually the loudest voice wins. Many approaches (driven by experience built belief) can effectively apply locally. Few, if any, globally. Understand the distinction. Trust your experience but always acknowledge and question your beliefs.

@kburman

This comment has been minimized.

Copy link

kburman commented Jan 31, 2020

Keep it simple, stupid. You ain't gonna need it.

“Good enough” isn’t good enough.

Aren't they contradicting each others?

@nVlast

This comment has been minimized.

Copy link

nVlast commented Feb 1, 2020

Also, write the specification of the system as executable code.

Could you elaborate more on that please? It seems very interesting.

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Feb 2, 2020

@eatkins Thanks for you kind words! I completely agree with all your comments.

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Feb 2, 2020

Keep it simple, stupid. You ain't gonna need it.

“Good enough” isn’t good enough.

Aren't they contradicting each others?

Not always, I think good == simple! And very often, I find that working on a bit of software to improve it entails making it smaller and simpler.

That said, there definitely is a tension between doing simple things that work well vs carrying on iterating to improve it, there is a risk of overdoing it when working on code, to the point where the extra invested time isn't worth it. Knowing when to carry on and when to stop is definitely a question of judgment, and one of those key skills that only comes with lots of experience I think.

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Feb 2, 2020

Also, write the specification of the system as executable code.

Could you elaborate more on that please? It seems very interesting.

In essence, it really just means that you should aim to have high level tests that describe the desired behaviour of the system - so you have a runnable spec that tells you what the system does, and if it is works as expected.

This isn't particularly radical, but I think that this becomes particularly powerful when combined with some of the other practices I suggest. For example, if you model your domain separately from backend implementations, and have simple reference implementations of things like data storage and communication between parts of the system, this enables you to test all aspects of the whole system quickly and efficiently.

@nVlast

This comment has been minimized.

Copy link

nVlast commented Feb 2, 2020

Also, write the specification of the system as executable code.

Could you elaborate more on that please? It seems very interesting.

In essence, it really just means that you should aim to have high level tests that describe the desired behaviour of the system - so you have a runnable spec that tells you what the system does, and if it is works as expected.

This isn't particularly radical, but I think that this becomes particularly powerful when combined with some of the other practices I suggest. For example, if you model your domain separately from backend implementations, and have simple reference implementations of things like data storage and communication between parts of the system, this enables you to test all aspects of the whole system quickly and efficiently.

Oh now I get it. Thanks for the insights!

@optevo

This comment has been minimized.

Copy link

optevo commented Feb 3, 2020

WRT "Keep it simple" vs "Everything is code".. Code is obviously not the simplest way to represent everything.

Anything that can be represented by data can be represented by code but not vice-versa and "code" itself has different levels of expressiveness: do I need a language that is Turing complete, or is a finite state machine or a regular grammar (of which Regex's are an example) sufficient? Or more to the point, when should I use each type of language? If data is sufficient, which data structures can/should I use? There's more than just JSON, XML and YAML in the world.

A useful example of where I am coming from is https://cuelang.org/ which has a great language because it tries to get the "power" balance right.

"simple" vs "expressive" are conflicting priorities and is would be great if you could elaborate more on how you might resolve those conflicts.

One principle I use is to use the simplest representation that works (i.e. sufficient to accurately model the problem space). Why? 2 reasons: It will be easier to share/communicate with others and simpler structures are more analysable than more complex ones. e.g. consider how easy is it to determine missing states in a FSM vs a program written in a general programming language. Or with the CUE language above, basing the type system on a lattice means that it's possible to statically analyse type changes to help determine if they are breaking.

I also have a problem with your assertion that "Use the smallest team possible, but no smaller". It's not that I think it's right or wrong; it's just not all that helpful because it leaves open the critical question of what is the smallest team possible and how do you know when you do/don't have it..?

@Fishbowler

This comment has been minimized.

Copy link

Fishbowler commented Feb 3, 2020

Interested in the intent behind

Absolutely all tests should be automated.

  • "We should endeavour to automate all the things that can be automated"? This still accepts that some things could be automated, but can't be because of cost.
  • "We don't ever need humans to run or even consider this before shipping to production because it's guaranteed effective by the automation"? This puts a level of onus on an engineer to have considered a problem from all angles, from scalability to user accessibility needs and everything else.
  • "Absolutely all functionality should be covered by automated checks"? Similar to the first one, but proves that the code operates exactly how the engineer intended, if nothing else, as well as calling out in test framework method names which environments and conditions that were considered as part of that implementation.
  • Something else?

Disclaimer: am a tester, so possibly hold specific meanings to certain words.

@primercuervo

This comment has been minimized.

Copy link

primercuervo commented Feb 6, 2020

This gist is a great read, and hits near home! Thanks for putting it together. Regarding the "Design" section, you make emphasis in "modelling", and this following sentence caught my attention:

Modelling - the act of creating models of the world - is a crucial skill, and one that’s been undervalued in recent years.

Do you happen to have some resources of choice for improving this skill?

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Feb 6, 2020

Interested in the intent behind

Absolutely all tests should be automated.

  • "We should endeavour to automate all the things that can be automated"? This still accepts that some things could be automated, but can't be because of cost.
  • "We don't ever need humans to run or even consider this before shipping to production because it's guaranteed effective by the automation"? This puts a level of onus on an engineer to have considered a problem from all angles, from scalability to user accessibility needs and everything else.
  • "Absolutely all functionality should be covered by automated checks"? Similar to the first one, but proves that the code operates exactly how the engineer intended, if nothing else, as well as calling out in test framework method names which environments and conditions that were considered as part of that implementation.
  • Something else?

Disclaimer: am a tester, so possibly hold specific meanings to certain words.

These are good questions! I definitely accept that there are some areas that can't be tested with automated tests, for example checking if we're building what the user wants, whether it's usable, and so on. However, in terms of whether the system does what engineering intends it to do, I think that's a responsibility that fully rests within engineering, and I find that this is something that has to be an inherent part of what developers do. Scalability for example, i think is something that developers should be acutely aware of - at least when needed!

I see having separate QA individuals or teams is often a symptom of problems in how the system is developed. For example: the system isn't easily testable except by manual steps (hence developers don't want to, as it's boring). Or the system requires a real environment to be tested (hence it's a bottleneck that interferes with development) - this is a symptom of a lack of abstraction in the system.

Don't get me wrong - in most development organisations, dedicated QA efforts are often needed in order to rectify problems elsewhere. But, I do believe that when things are done well, the need for this largely falls away.

One caveat as well: I'm talking about this in the context of backend and data processing systems. There may be other areas (front ends, mobile apps, embedded software etc) where things may be different.

@gihankarunarathne

This comment has been minimized.

Copy link

gihankarunarathne commented Feb 6, 2020

I don’t care if you write the tests first, last, or in the middle, but all code must have good tests.

Prefer to start with test, because it gives the confidence about the code. No need to test all the code, but the critical points at least.
Basically, it'll take more time at the beginning and you feel like wasting time. But later you'll save that time while avoiding waste your time on debugging. Like mentioned by @Fishbowler, it comes with some cost. After writing test cases, you have to maintain them. If you're a confident programmer and based on the few other factors, may be you can skip some part.

@ardityawahyu

This comment has been minimized.

Copy link

ardityawahyu commented Feb 8, 2020

Out of all the agile practices commonly used, estimating task sizes and trying to measure project velocity is the least useful. (Its utility is often less than zero).

so how do you plan the project? This usually needed for the higher stakeholder

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Feb 8, 2020

Out of all the agile practices commonly used, estimating task sizes and trying to measure project velocity is the least useful. (Its utility is often less than zero).

so how do you plan the project? This usually needed for the higher stakeholder

In my experience, gut feel is better than any system of calculating story points and trying to linearly extrapolate that to future work.

But any attempt at estimating software projects is bound to be very inaccurate. So I'd say the best approach is to educate stakeholders to accept this uncertainty, showing them that you can deliver incremental value, and let them steer the direction in choosing the priority of upcoming features.

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Feb 8, 2020

This gist is a great read, and hits near home! Thanks for putting it together. Regarding the "Design" section, you make emphasis in "modelling", and this following sentence caught my attention:

Modelling - the act of creating models of the world - is a crucial skill, and one that’s been undervalued in recent years.

Do you happen to have some resources of choice for improving this skill?

Thanks for the kind words @primercuervo. And that's a very good question!

The Domain Driven Design community is a good source of material on domain modelling. See for example Scott Wlaschin's Domain Modeling Made Functional.

Otherwise I struggle to find good texts that teach the art of modelling. As I say, I think this is an area that doesn't receive enough attention! I mostly learnt what I know about this the hard way, through work on various projects. I also spent quite a few years working with semantic web technologies. The book Semantic Web for the Working Ontologist: Effective Modeling in RDFS and OWL is a good one for this area, but is probably too specialised for general purpose modelling.

It's probably worth looking back in time to older sources for inspiration. "Data and Reality" by William Kent (published in 1974!) is a great source of information (if not a how-to book). Books from the database design community may be useful, but I don't know any of theses myself.

This isn't a very satisfying answer I'm afraid - If anyone has suggestions here I'd be very interested to know!

@manawardhana

This comment has been minimized.

Copy link

manawardhana commented Feb 12, 2020

Wow, This is an excellent compilation of a library of books into a couple of pages. Each line can be expandable into its own book. Thanks for sharing. I believe every developer/architect should revisit this time to time.
On

Resist the temptation to use your main domain types to describe interfaces or messages exchanged by your system. Use separate types for these, even if it entails some duplication, as these types will evolve differently over time.

I know the above point is the accepted norm and it is safe. I don't disagree.

But this is one of the most boring piece in Software Development. The alternative is NOT using domain objects for messaging. We have to rethink the relevance of our habits of using types and OOP which is a separate discussion.

GraphQL might have solved this to some extent (not for everybody nor everything). But I guess there are more alternate approaches.

@erebos12

This comment has been minimized.

Copy link

erebos12 commented Feb 14, 2020

Something I would add to your list:
Fundamentals: Don't repeat yourself! ;-) Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

@GrayStrider

This comment has been minimized.

Copy link

GrayStrider commented Feb 18, 2020

if you’re writing JSON or YAML by hand, you’re doing it wrong

Can someone comment on this? I have my configs in JSON and YAML, how would one do it better? Or am I misinterpreting the message here

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Feb 18, 2020

if you’re writing JSON or YAML by hand, you’re doing it wrong

Can someone comment on this? I have my configs in JSON and YAML, how would one do it better? Or am I misinterpreting the message here

Sure thing @GrayStrider! My view on this is that for genuine configuration, i.e. settings that vary between each installation of a system or service, storing these in config files in whatever format is fine. This really should be a very small number of settings though.

What I see a lot of, is that people use "configuration" files to do much more than this, for example describing the cloud infrastructure that their system runs on, and how different components inter-connect. For large systems, this can result in many thousands of lines of YAML/JSON or similar (I've seen cases of six figures!). This is when it gets very problematic. JSON or YAML are clearly not very expressive, they have no type system, and very poor mechanisms for sharing common code and building abstractions. This leads to lots and lots of duplication (copy & paste), for example.

Sometimes people try to alleviate these problems by building on top of the JSON/YAML formats, for example using templating engines, or adding basic facilities for variable substitution and so on. These are usually terribly clunky, and a far cry from using a real programming language. And when was the last time you saw unit tests for such configs anyway?

What I would much rather do, is describe all aspects of my system as code, using a high level, functional language with a good type system, in a high level domain model. This description can be interpreted to produce concrete implementation, either in code, or in the config file formats needed to make things happen (e.g. Cloudformation templates, Terraform scripts etc). Apart from saving me the pain of writing JSON or YAML by hand, this also encapsulates implementation details about the infrastructure, which makes for a system that's much easier to maintain and evolve over time. And it can allow multiple implementations, e.g. for running a system locally vs. on a cloud provider, or it can support multiple cloud providers, and so on.

But as I said, most people don't do this (yet), so don't feel to be if you don't either... :)

@GrayStrider

This comment has been minimized.

Copy link

GrayStrider commented Feb 19, 2020

Would love to take a peek at some open-source examples of this misuse, @stettix, if you happen to know of any. I had no opportunity to work on such large systems yet, but indeed, it seems only reasonable to convert the configs into more dynamic formats once it's clearly needed

@stettix

This comment has been minimized.

Copy link
Owner Author

stettix commented Feb 23, 2020

Would love to take a peek at some open-source examples of this misuse, @stettix, if you happen to know of any. I had no opportunity to work on such large systems yet, but indeed, it seems only reasonable to convert the configs into more dynamic formats once it's clearly needed

@GrayStrider : I don't know of any such application I'm afraid. This is a common problem: most open source projects are libraries, frameworks and middleware, I don't know of many examples of actual applications and real systems of the type that exist around every enterprise. So it's difficult to see what such code looks like without actually working for lots of companies!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.