I've heard that the best devs write tests that are born to fail. Then they implement the feature to make the test pass. Test-driven development (or even documentation-driven development if you're really into it) is a great way to write good code. Apart from forcing you to do your least-favorite part first, TDD also makes you take a step back and really think about what you're trying to implement. What are you really trying to achieve? What are the side-effects, what are the inputs and outputs that you will need to mock?
This is all well and good, but one time I feel like TDD doesn't actually work all that well is in the very early stages of building your greenfield project. In my experience, TDD leads to more work refactoring a nascent project than you save by automating the tests. If you barely have an idea how your project will be structured, let alone the code that fits into that structure, you're going to end up re-doing a lot of work. However, there's no hard and fast rule for when you need to stop cowboying around and start automating your testing infrastructure. Here are a few things to look out for and a few tricks along the way.
You can get around this one with some shell aliases early on, but keep them short. Avoid the temptation to write any shell scripts, those are just lazy versions of your language of choice's testing framework. However, feel free to write some proto-test cases in whatever language you're using e.g. a scratch.py file. You'll likely be able to scavenge it when you write proper tests.
This one is fairly self-explanatory. Once the overall structure has ossified a bit, you can feel more comfortable committing to a testing directory without fear of breaking all your paths in a couple commits.
If you have to go back and do some digging to figure out which entrypoint or config to use to test a certain way that you expect your app to behave, this is a good sign that writing some proper tests is overdue. The good news is that writing your tests will help quell that complexity.
Search through the tools you depend on to see what their testing infrastructure is like. Any mature project worth it's salt will have a whole slew of tests, and a whole slew of opinions about how those tests should be structured. They might even have utilities for bootstrapping your test cases. I'm using Snakemake to create a data processing pipeline, and they provide a wonderful utility for automatically generating unit tests. I'm also building on Golem and they have a whole test harness for their apps with a plethora of examples of how to write good tests.
Those auto-generated tests I mentioned above were great - but they generated a lot of boilerplate. Look for opportunities to refactor those common functions/classes into something reusable and extendable. I was actually running my app in a container whereas the tests were made to be run locally. I decided to make a TestRunner class which can be composed of different Runner classes and output Checker classes. Work with what you already have, but then make it your own to best suit your needs.
There's nothing wrong with using a hack here or there to make something work. It might not be the most elegant solution, but if it works you can move on. The benefit here, especially with a new project, is being able to move on to what you're actually trying to achieve. You'll be able to see the bigger picture sooner, allowing you to go back and refactor that hack in the best way possible. Perfection is the enemy of good - it's also the enemy of finishing anything.
I'm writing this shortly after getting all the tests to pass for the features I currently want to cover. It was not my favorite part of this project so far. Oftentimes I found myself chasing down a bug only to find another bug, or worse - a show-stopping design decision.
One such example was having to refactor my Docker image to be rootless instead of running as root like it's base image. My tests involved mounting a volume from host to load the mock inputs and check their outputs. Since the image was being run as root, then any outputs had root permissions - so my (non-root) test runner would get permission denied errors when trying to clean up! I (accidentally) worked around this issue by moving my testing assets onto the NFS share that I was mounting from my host into the Vagrant vm. Anything written to that share was getting it's permissions coerced back to 1000 from root by, I'm assuming, vagrant or libvirt.
The moral of the story is that this unpleasant testing experience forced me to prioritize something out of the backlog that really ought to be prioritized (rootless container) and helped me learn more about my dev environment. This experience hasn't just been about writing my project's first tests, it's also been a visceral experience with tech debt. A little corner-cutting goes a long way towards having a higher understanding of what you're building. Too much and you'll end up regretting it dearly down the line. I certainly have a lot more tests to write (and re-write) - but the opportunity to better understand my project and where I should focus my efforts has been well worth the struggle.