You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This document serves as a table of contents and map to UC San Diego’s CSE 110 - Software Engineering as taught by Thomas Powell for Spring 2023. Offerings by other instructors may vary in emphasis and content so use this material with extreme caution if somehow you have gained access to it outside the course.
This is a work in progress. I'll change this update statement when I make a major change
Extra Item: Goofus and Gallant Cartoon Stories - this had some small tips about good and bad behaviors.
Extra Item: Animal Friends Introduction - this used vidid images from animals to show encouraged behaviors for teams. It was meant to show the commonalities of teams before we introduced the topic formally.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lecture Notes - SE Intro - Demystification, Definition, and Defiance
Premise - First We Must Understand Our Definition and Biases
Software Engineering is defined in numerous ways. It is quite interesting that for a discipline focused on engineering precision and a scientific process it is a bit problematic there is such variability. The goal here when introducing SE is to show you the variability, create our own working definition and most importantly explore our pre-existing biases about the topic. Very often for those of us learning computer science and programming first in an academic setting we have a strong belief that the activity called Software Engineering is primarily code focused when in reality it is more a socio-technical process which code is maybe 30-40% of the emphasis. Breaking that bias down and understanding the essence of engineering beyond SE is a key point of the introduction.
Key Support Points and Ideas
Why So Much Fail and Worry?
A number of statements from various important SE folks like Alan Cooper, Fred Brooks and others show that even the most advanced SEs are more than a bit worried that our engineering rigor and ability to find constant rules are more than a bit lacking.
Cooper is cynical enough to think success is almost an accident.
Numeric costs of trillions of wasted dollars are cited just for the US alone
The Professor has personally seen organization spends millions for things that can be accomplished for a thousands. Further we have seen that even successful projects are lacking engineering formalism even in some of the most financially successful companies. Given the security failures, data leakages, and out right crashes it isn’t difficult to see the fail.
The question that is open is why do many people with significant experience worry and why is there so much observable fail? This cannot be settled in this course, but the aim is to arm us with tools to find our answers and get on a path towards more rigor and less luck.
Definitions Vary
Per Wikipedia: Software Engineering is the study an application of engineering to design, development and maintenance of software.
Per the definition above Engineering is key to understand.
Engineering is defined as a creative application of scientific principles to design or develop software with awarenessof the design, to perform intended functions and understand and forecast the behavior of the system under specific conditions with some emphasis on economics and safety.
More simply put - use science to build something that we know why/how it was done, can reason about it under normal conditions, it does what it should and is done economically and safely!
Other definitions try to be a bit simpler
Parnas: SE is the multi-person construction of multi-version programs
Farley: Software engineering is the application of empirical, scientific approach to finding efficient, economic solutions to practical problems in software.
Also Farley States (Paraphrased): Software Engineering = Design Engineering + Production or Process Engineering
I tend to prefer Farley’s break-up and the alternative view above as it shows the more creative side and more constructive side of the pursuit as quite different. The production engineering tends to share more rigor and process like other engineering disciplines while the design portions tends to follow more design thinking and creative processes. The actual mixture of emphasis is likely to vary depending on project and situation. For example, start-up heavier on design engineering, critical and mature system more on production engineering.
Testing and interview wise you should be able to succinctly define SE in your own manner hitting some of these points. Also you must accept that clearly if so many luminaries can’t agree on a definition it might be quite a bit fuzzer that some sort of “physics of programming” I personally want a very concrete definition, but that lack of is a key problem for why we often don't succeed in SE as what matters varies from person to person, organization to organization.
The Topic is Huge
A significant reason we have definitional trouble is that the topic is broad and deep. Compare SE to say Skyscraper construction. In a large scale construction project we have physics and structural engineering, materials, architectural, economics, various subsystems from electrical, hvac, etc., project management, regulations, safety, interior design, and much more. Software is just as varied and each of the knowledge areas (KA)s a total of 15 of them is a massive topic
It is unrealistic to expect to master SE in a short period of time like a quarter or semester. This is likely a career level process and some individuals may only focus on a particular knowledge area as they are that broad. I purposefully expose you to this in the course because I think being more constrained and focused absolutely sets you up for a certainty you simply shouldn’t have so early in your study of the topic.
Test Note: Please be able to roughly identify these topics. Two of them Computing Foundations and Mathematics are not covered in the course and are assumed knowledge at a basic level. Engineering Foundations is weaved in constantly as opposed to a specific topic. The other topics are touched on in varying degrees as the class unfolds with the most emphasis on Requirements, Design, Testing, Process and Methods, Professional Practices, Quality and Testing.
Beyond Test: Note that the field is simply too expansive to be a deep expert in ALL aspects of SE.
SE >>> Coding
The previous section on SWEBOK should clearly show that coding is just a portion of the SE activity. The amount varies depending on your role and the organization, but coding generally makes up below 50% of the activity and often far less in complex systems.
This shouldn’t be debatable really, but our biases with significant learning emphasis on coding may clang with this thought. A few examples of large organizations that state this reality were provided by Google to dispell some idea that it is different if you just get to some fabled land of engineering.
SE is Less Technical and More People Oriented
Following along the SE != Coding idea, SE deeply focuses on people issues be it end user or team focused. Amongst senior engineers this is not at all debatable, amongst junior engineers it can be quite triggering to the point of completely disbelieved. Plenty of folks have said this over the year such as DeMarco (Peopleware) and many others. A few supporting statements are provided to show this reality goes far beyond the Professor who subscribes to this so much that I believe that is the true cause of why software projects fail.
Embracing Our Biases
If previous iterations of SE courses is a guide some of these introduction points can be quite troubling to those who want a precision and predictability implied by the coding tasks they have encountered in an academic setting as well as what the word engineering implies.
I believe that if you understand that you may have biases about what you believe the topic to include you’ll find your way to a good definition and emphasis that serves you well. Moving from a fixed mindset to more of an open growth mindset is useful.
As mature engineers we should have strong opinions, but also hold them weakly especially if new information changes our view.
Are SEs Really Engineers?
We read a paper about the loose application of engineer to the activities we perform and professional posture we preform them under. I would agree that the essence of engineering is often missing and many with the SE title are actually just coders. A professionalization of the SE space may be required to address our failings and improve our industry. I am hopefully that may happen, but for now we start with just admitting at least some of the criticisms leveled at SEs are at least partially true.
To address this challenge we performed an “Individual Engineer Statement” activity to define what our SE aims are and take more formal control over our working definition and exceptions about engineering. Likely that definition will evolve over your career or even in the class itself.
Test or not you should have a short version of your working definition of what engineering means to you and what you are planning on doing to improve yourself to live that defintion.
Conclusion
It is hard to talk reasonably about a topic if the definition is unclear. Sadly the definition of SE is in fact somewhat unclear and I posit it is still evolving. We spent some time defining the topic the best we can, seeing the overlaps, and admitting that it is a bit of a moving target. We can say that SE is certainly much greater than coding. However, its’ size is also a significant part of the definition problem is the sheer size of the topic - it’s huge (15 areas!). We also need to acknowledge our biases that grow both in how we experience learning about computing as well as social media or industry beliefs. Being aware of our biases and being willing to think about strong opinions and humorous criticisms may serve as a useful inoculation against a dogmatic view of the topic.
Epilogue - Why the Memes Matter
If we can’t poke fun at ourselves and our industry we are unlikely to be able to have the safe conversations necessary to take a hard look both at what we are doing well and quite poorly. The Meme portion of the introduction lecture had numerous examples of fails and absurdities about SE and learning SE in an academic setting. The vast majority of these were NOT MADE BY THE PROFESSOR. While I certainly collected them I make this point explicitly as it shows you than quite a number of people seem to encounter the more than occasional SE absurdity and they made Memes as a way to cope or share their frustrations and add a little levity into a challenging environment. In general that is a useful tactic and being able to take yourself a bit less serious especially as things go wrong can be a valuable skill. In conclusion these ideas are not very unusual and if you feel they are wrong you might want to use them as a tool to reflect upon your own biases and beliefs.
TL;DR - the memes matter and may remind you of key ideas or failings, but they are more to prompt you than to be studied for a test or class project.
Test Questions
Defining Software Engineering in Your Own Words
As there appears to be no exacting definition of SE, please attempt to explain it yourself using common definitions and how you understand it. Try to include the ideas from the class, Wikipedia, etc. for full credit.
Why So Many Definitions?
Now that you have defined SE in your own words and acknowledge the varying definitions explain why there are so many variations. Hint: Consider breadth of topics, type of projects, risk & impact, practitioner background, and more.
To Professionalize or Not?
We read a provocative paper about if Software Engineers are engineers or not. This prompted us to write a personal statement and make a plan to understand what we wanted to be, but in this question we meet this professional engineer point head on. Do you think it is useful for us to professionalize, to have certification, standards and practices, and so on? Explain why you think this is a good idea and bad idea. The best answers may argue both sides to show deep consideration.
Besides the ability to learn and apply technology, it’s been widely shown to truly be successful at software engineering requires an open mindset, supportive habits, and significant discipline. The practice of software development is just as taxing if not more than any other domain and to suggest that we do not need to focus on such things is to suggest other domains simply are all wrong. From sports to cooking to just about any space you can think of some significant emphasis is placed on aspects outside of the skills of that space, and this segment attempts to do the same for software engineering. With these and many other mental tools in place we are on our way to grow from a novice developer to a senior software engineer given enough time and energy.
Key Support Points and Ideas
SE Productivity is People Driven
The most important factor in improving software development is not the tools and techniques used by programmers, but thequality of the programmers themselves.
Other indications fromDeMarco (Peopleware), Fred Brooks, and many others which posit the main issues with software projects are often sociological and not technical. This is backed up by the Prof's personal experience over 20 years.
Economic studies suggest the same
10x Developers Myths and Reality
The idea of a high performing developers is often short-handed as a 10x developer as the productivity and output of a highly performing developer may be as much as 10x of an average.
The idea of a 10x performer is not unique to SE, consider any field such people exist (ex: in soccer the idea of a superstar)
Becoming a super start in most professions is a long long journey that involves significant commitment and practice. The idea that the youthful dev is some digital native that immediately exhibits significant competency beyond their peers is a myth more than a reality. Even amongst sports or other professions the wunderkind generally had significant training you simply aren’t aware. TL;DR - the natural born 10x makes a great story but it isn’t realistic.
10x developers often can cause 10x trouble because when coupled with a poor personality they can make a team underperform rather than over perform. Again this type of thing is no different from the moody lead singer of a band that rips it apart of the touchy star player that doesn’t motivate a team.
Obviously we might want to aim to become a 10x developer, and try our best to omit some of the negative aspects.
Mindset - Postel's Law of SE Professionals
Step 1 on a 10x generation is starting with the right mindset. There are things we can control (ourselves) and things we can only influence (others, our organization, society, etc.). Postel’s law also known as the robustness principle drives much of how the internet works. Roughly it states be permissive in what you accepts, and strict with what you emit. This seems like a good plan in dealing with what we do (be strict) and what we encounter outside (be permissive)
Motivation and Mastery
Intrinsic and Extrinsic motivation is important to distinguish.
An external motivator might be a paycheck, a super nice car you want, a new job title, etc.
An internal motivator might be the enjoyment you have when you solve a puzzle, how it is interesting to explore a new technology, or finish a project.
It is my general view that external motivators are harder for you to control (think Postel’s Law) while internal one’s you can. For a long career you probably need a bit of both. However, do not misinterpret as a value judgement. I am not saying one is better than the other, just that you likely need a bit of both. Your personal balance will likely not match anyone else.
Finally to stick with something like SE long enough to approach a mastery level you need to separate attitude from ability. Likely you will always be learning and trying to get good at something new as your ability at a new skill won’t grow out of nothing. You need to have a positive growth oriented attitude first and put in the effort to grow your skills.
Generally I often will say you can train a skill, it is tough to train an attitude. A person needs to adopt the attitude first.
Practice Makes Perfect Better
Following along the previous point we need to put in appropriate practice so we mastery a skill. We should assume that we will unlikely get “perfect” at something with practice, but quite likely we will get better at it with appropriate focused and consistent practice.
This relates later to our dev habits where we break out some time to practice a skill rather than just immediately expect to be good at it.
Learn and Practice Both Tech and Soft Non-Tech Skills
Practicing skills isn’t just about things like typing, learning a coding syntax, IDE macros, etc. but also can cover so called soft-skills.
I rename this non-tech skills rather than soft as that implies some hierarchy of importance. It turns out both types of skills (tech and non-tech) are important.
Examples of practicing non-tech skills might be trying to speak up more in a meeting, writing for fun such as blogging, working on an organizational schedule for your days, developing habits, etc.
Good Gear Can Help
Good “players” can still succeed with bad gear and bad players don’t instantly become good ones with good gear, but it does help.
As we play the game of Software Engineering it is important to consider our gear. We really must first focus on our input/output as that is how we get our thoughts into code. This suggests keyboards, screen, and mice are pretty important.
Generally my attitude is that we try to get the best gear we can that we are comfortable with. Be careful with replacing gear all the time thinking it will make you better, but invest in yourself not just with your habits but with your devices as appropriate.
Industry Tip / Observation - good orgs don’t skimp on equipment. It is an easy way to get higher performance out of devs to make sure they have what they prefer. If necessary, bring your own gear and advocate for more quality “kit” so to speak. Note this idea is not unique to me at all Spolksey and others have talked about this for well over 20 years and most larger tech companies like Microsoft have traditionally being supportive of devs in this area.
Common Productivity Myths
Constant sprinting - you can’t expect to be optimal all the time and perform at maximum effort
Rest and efficiency - like the previous statement you need rest to be efficient. You might be able to hack energy for a while with Redbull, but ultimately a few hours with appropriate rest often beats 10 hrs with little rest
Flow State - when you get in your optimal state of work it is almost a feeling of natural flow. It can be very difficult to get into this state without appropriate rest and minimal distractions/interruptions. Guard your ability to get into this mode as it is key to quality software development
Multitasking and Distractions - an enemy of flow state is distraction. From an interruption or self created distractions it is really impossible to be great at your dev tasks if a bunch of other things are stealing your attention cycles. The thought we can multitask and that is efficiency is also a massive fallacy many subscribe to. The science shows you can’t really do it at least not well so try to avoid it if you can. We see this type of idea manifesting all over software engineering for example the idea of keeping your WIP (work in progress) low, to work on a task until done, etc. are all supportive of this base idea.
Rhythms - understand your own rhythms and accept them. If you aren’t a morning person don’t code then for example. Admit when you are tired and do less stressful tasks, basically admit to our “daily seasons” and act appropriately.
Avoid Yak Shaving and the appearance of busyness - many in our industry are very busy doing meta work. Traditionally in software engineering we often talk about the type of work with think is necessary or have to do before real work as “yak shaving”. Try to limit falling the track of appearing to work. You’ll see many examples with Github punchcard faking, constant social media proclamations, configuration or cleaning tasks, and more once you start looking for performative hustle.
Attitude Challenges for SE Growth
Confusing Confidence and Competence - be careful with faking it until you make it
Impostor Syndrome (aka Comparisons and Self Esteem) - like item above we do know when we are faking and if we see others doing we can deeply harm our self-value. The idea we are an impostor is a dangerous thing and it has been significantly amplified by a social media powered comparison culture. Remember we all don’t know things and experts were certainly novices at one point or another
Absolutism (aka Black and White or Binary Thinking) - when it comes to humans which includes us we need to be very careful with absolutist thinking which is common amongst engineers. Example is X is the best, Y is the worst. The truth is that thing being best or worst is pretty subjective and given condition often “it depends.” Applying strict absolute thinking to our behaviors can be quite toxic
You != Your Code - it is important to have pride in our work or code. However, taken to an extreme we may personalize our work too much. The truth is that even the most gifted developer will write some bad code. Try to avoid tying in your self value solely with your work or code. Commonly it used to be a common thought in engineering to be dispassionate about things so you can just allow something. In other words, we aren’t so worried about what we did as it is external to us. Unfortunately that is less broadly held amongst developers today.
Selfish and Selfless Devs - selfish devs often are quite worried about what they think and how they feel. This often leads to making decisions that may harm others including end users. Likely that is not one’s intent, but we should be a bit more selfless and understand who we do things before.
The T Shaped Dev
A developer with too much breadth but no depth isn’t great except in overview situations. Such folks often do better as project coordinators or in small orgs where we can afford to do less quality with less people. A developer with too much depth but lacking breadth often misses important external effects or opportunities. These developers also are more valuable in large groups as specialists, but may run the risk of being eliminated if the speciality falls out of favor. The best devs tend to be a “balanced T” and have breadth and depth within reason.
Good Dev Habits
Good dev habits vary, but start simple like showing up and being present. For example, if you go to a meeting, be present in the meeting not checking your email or zoning out. Habits will vary, but I have found things like small consistent progress over time as a key one. Basically try to aim to do a few steps on something important to you as often as possible without fail to get big huge gains. What is important to you I can’t say, but it might be anything from learning to use your IDE better, to understand many algorithms, etc.
Allowing Failure for Growth
Many developers suffer from a need to be a perfectionist. This is a bit of a variation of black and white or “binary” thinking where we either perfect or a failure. The truth is we are often all a bit good and a bit bad at what we are doing. Our aim should be to allow ourselves to fail a bit and just work at things with less judgement. We must reserve judgement as growth only comes from when we push ourselves and that will result at times in failure. If we can recast a failure by seeing what it taught us and keep the effects of the stumble minimal we can effectively grow
Walking the Path from Novice to Senior
A number of tweets and quotes were provided to show various tips on getting on the path to a senior engineer. Interestingly many of these tips have more to do with skills beyond tech. There was a bit of sameness to the tips, but the truth is the path is long and winding. You’ll need to be patient and know that how long it takes and what you need to work on will vary, but stick with it long enough you’ll get there! I’ll jokingly (not really) say that you need to “Train for a marathon, not a sprint” in our field.
Conclusion
It’s been long held that the biggest factor for improving the quality of software and the success of a software project are the developers themselves. So called 10x developers or more realistically high performing developers do exist and can significantly help or hurt software teams. Unfortunately building such developers isn’t some simple algorithmic process. A key step is adjusting our mindset to growing, understanding our motivations, working at our habits, accepting that we may need much more than technical skills, and be willing to stumble along the way. While much of this topic might be new to you in your CS program little of it is really that new. In some cases I just recast materials, in other cases I took SE observations commonly held. Regardless of how/where I got the ideas of the segment the quotes and books significantly back up the importance of these tips so ignoring them in favor of some more predictable tech thought may be quite impactful on your career growth.
Epilogue - Accepting We Just Aren’t Deeply Different
Again many of these ideas seem to involve a value judgement upon developers. We produce product of the mind and often to do it well we need heavy concentration and flow. Many of the ideas pushed at us unfortunately are counter to our actual success. A grind or hustle culture epitomized by the idea of 996 or the start-up culture is a near gurnatee of burn-out. Like an athlete over training, getting injured, or just not reaping the benefits of effort so to do we find individual devs defying human realities.
It truly makes no sense to suggest that we don’t need to listen to our body, to not address our attitudes, to overly push or aggressively pace ourselves, and to have a wholy unrealistic expectation of our ability and growth. If we were athletes we’d train constantly, assume we would never lose, if we lost blame others, attempt to sprint a marathon and somehow take our growth curve to believe we could run a 2 minute mile. Even worse we’d likely believe Nike it they claimed their shoes could help us run that 2 minute mile.
The silly comparison given is just one technique of the segment to knock us out of an unhealthy views that suggest everything is first mover and winner take all at any expense. The truth is that software already has more than 50 years of history and such thoughts have more often lead to burnout and failure in the long run at best the hustle culture offers for most a short term win giving way to a long term loss. We sadly build around the few outcomes assume that we will somehow rise above. Rather than fall for the fairy tale, maybe we should enjoy a process that allows to see that we are not deeply different than other humans and if we can accept that we will likely be better devs and work better with others.
Example Test Questions
10x Developers
Explain the idea of the 10x developer in Software Engineering. How does one typically become a 10x developer? Discuss the pros and cons of a 10x developer.
Tools over people
Some people deeply wish that tools and technology will solve developer productivity issues. Data and history doesn’t seem to suggest that is a good bet. If that is the case why do you think people keep pushing the tool over people idea?
Provide a modern example of this thinking and at least one thought that may make holding out for a big tool/tech win unlikely.
Balanced Engineers
What does it mean to be a “T” Shaped engineer? Provide a short example of how a T shaped engineer might save a project or team.
Psychology Challenges for SEs
What is impostor syndrome?
Why do you think developers get impostor syndrome so commonly?
Valuing Failures and Resiliency
If failure is required to learn things describe how to approach failure in a safe way as a learning dev. For maximum consideration make your answer personal or related to what you experienced on your team or an internship.
Industry and Societal Challenges for SEs
At times it feels like industry is moving so fast that anything you learn in school or even on a job is out of date before you even master it. Figuring out coping mechanisms to address the whirlwind of change is a key skill for a software engineer these days. Describe how you might approach this for your growth beyond this course. Hint: Think about dividing your learning efforts and time % wise and topic wise.
Explain Individual SE Misconceptions Using System Thinking
SEs are smart people and like most people we do things for real reasons. However, as we have observed in this segment and across the industry there are many misconceptions and beliefs that seem somewhat troubling as we stop and ponder them. The Prof contends that some of our beliefs and misconceptions are likely driven by numerous forces many potentially more system based including economics, social, regulatory, custom, convention, and more. See if you can think of an example to support his thoughts. Avoid just saying someone is ignorant. If ignorance is a key component explain how that came to pass from a system’s point of view.
Premise - Software Engineering is a Multiplayer Sport
Meaningful software is so large is simply must be produced by a group of people. However, getting a group of people working well together can be quite difficult. Group dynamics are certainly not unique to SE and we see techniques to make high functioning teams from everything from sports teams to kitchens. However, SE has specific aspects in terms of our culture, beliefs and attitudes that should well considered. Interestingly, though the data collected after much studying suggests that the sameness might overwhelm the difference so I encourage students to be a bit broader in allowing SE group learnings from many disciplines.
Key Support Points and Ideas
Why Teams are Required
Team Formation Steps
Team Size and Communication
Team Composition - Same vs Varied
Team Composition - Star Players Benefits and Challenges
Team Topologies - Relating Org Ideas with Teams
Team Leaders
Some of -ilities like security, reliability, etc. are not user focused, but many are including:
The system provides the required functions - Utility
Ability to access the systems and its function - Availability
Ability to access the systems within acceptable time - Performance - ility :-)
Ability to be able to use the functions - Accessibility
Ability to be able to use the functions successfully - Usability
Ability to enjoy the functions - Satisfaction (Satisfiability)
These -ilities are architectural decisions that have to balance costs and some of them are at tension meaning that we might not be able to do everything in some perfect way (aka we must except trade-offs here)
Team Challenges - Generally
Team Challenges - SE Specific
Conclusion
Meaningful work requires teams, unless we are very willing to extend time the iron triangle of the next segment forces us to add resources and with such an addition comes challenges. There is overhead with teams in the form of communication and coordination. Add to this that we humans are not interchangeable resources, but unique people with a wide range of abilities from 1x to 10x so to speak. We quickly find that IQ will not make up for a lack of EQ with teams and even the most technical proficient team may devolve into a mess. While SE has some specific aspects, a group of humans working together on a shared cause is fundamentally not a unique concept and we should first aim to understand the broad concepts of group dynamics and see the particular software and software industry nuances second. Simply put, to get better at Software Engineering in groups we should admit that as software practitioners we might be spending a bit to much time on the procedural how and not the who of our constructive efforts.
Epilogue - Solo Artists vs Teams & Why Do We Resist?
Again many of these ideas seem to involve a value judgement upon developers. We produce product of the mind and often to do it well we need heavy concentration and flow. The need to interact with others also may not come natural given the types of folks who find such problem solving motivating.
Our industry does continue to promote the idea of brain power trumping social understanding especially in the light of the hooded hacker alone slaying a hard technical challenge. This heroic view of our work is in fact mostly a myth, representing special cases often from longer ago when software wasn’t quite as ingrained in our society requiring it to be done at scale.
In short, our resistance to teams often comes from a lack of shared positive experience with them, such as what might happen in this very class coupled with the control we can exhibit as a solo developer. Further, we might add into this a reasonable unfamiliarity we might have with such soft or social dynamic skills, but it is of my opinion as well as the majority of academic and industrial SE community that these are important skills and we very much can learn them. The first step though came from the previous segment were self-mastery must be achieved before moving on to group mastery.
Understanding your particular users and their needs is the key to the design engineering part of Software Engineering. Even if you do not have a role in this part of the process, keeping user in mind will help you may better technical decisions as you will be considerate about how your choices might effect user success and satisfaction.
Key Support Points and Ideas
Defining UCD
We call a strong emphasis on user during the constructive process - user centered design or UCD
I often tend to think of this these days like moving from a earth centric to a helio centric viewpoint as moving from a tech centric to a user and user centric view point.
Key UCD Laws and Tips
Law: You are != your users
Law: Users cannot be your designer
Tip: Avoid asking users what they want, instead infer it and verify it
Tip: Avoid “spooking the animals” if you decide to test or interview
Tip: You can’t be everything to everyone, there will be no perfect usability, a11y, etc.
Engineering pragmatism requiring you to balance trade-offs and to define what is "good enough"
Sampling of UCD Techniques
UCD focused thinking
Persona generation, User Stories
Observation
Direct
In-Direct - Analytics
Interviews
Avoid guided questions
Listen
Aim for free form responses
Common UCD Artifacts
Personas
Beware of devolving personas too much into stereotypes
Consider your “mom” or other real people to avoid this challenge
User Stories
These are considered Agile concepts
As a <blank> I want to <do blank> in order to <blank>
Customer Journey Maps and Other Scenario Diagrams
Try to understand that your software lives in your user’s world and is not their whole world.
Their steps may occur over time and offline as well - most software needs to meet the world not be the world (better tell Zuck!)
From UCD Research to Requirements
Figuring out what they want is the first step which is then followed by making it work for them which has a range of -ilities. Both of these processes should happen before code is done and be documented in the form of design documents often ADRs (Architectural Decision Records) as well as the artifacts mentioned above (ex. Personas, etc.)
Broad Requirements - Defining -ilities
-ilities are often what may call Level 0 decisions which set some broad requirements a system must aim to adhere to. You should be careful working on projects that have not done the work to define these as once their effects are felt by actual end users or result in negative business outcomes, then people tend to come to look for those who “caused” it and they may view we SEs are to blame - even if in some sense this might be above our pay grade.
TL;DR - Demand or define yourself system -ilities or put yourself or a project at a big risk
User Focused -ilities
Some of -ilities like security, reliability, etc. are not user focused, but many are including:
The system provides the required functions - Utility
Ability to access the systems and its function - Availability
Ability to access the systems within acceptable time - Performance - ility :-)
Ability to be able to use the functions - Accessibility
Ability to be able to use the functions successfully - Usability
Ability to enjoy the functions - Satisfaction (Satisfiability)
These -ilities are architectural decisions that have to balance costs and some of them are at tension meaning that we might not be able to do everything in some perfect way (aka we must except trade-offs here)
User Thinking Must Tie to System Thinking
Danger: You must tie your user thinking to your system thinking, if you don’t you will likely bake in a critical flaw that may hobble user acceptance.
The Importance of Accessibility
Accessibility is quite an important one because it can be discriminatory along physical ability as well as social position (think lack of hardware or connectivity)
Accessibility (a11y) is not limited to just extreme cases like blindness and may be temporary or situational
Choices we make with technology can effect most all of these issues and sometimes inadvertently.
Example: non-semantic HTML is inherently inaccessible as it provides no hints to assistive devices. While it can be rectified your <div>-itis can catch up to you and may even result in legal liability
Conclusion
User Centered Design is a key skill for developers to differentiate themselves from coders. We should be extremely focused on the -ilities which we often call non-functional requirements and how users focus on those. Knowing key laws and familiarizing yourself with common UCD thinking should allow you to tie user needs with technical implementation.
Epilogue - Why Does This Go So Wrong?
Many of these ideas seem to involve a value judgement upon developers. This is no a negative intention and quite often such choices are out of ignorance of the effect of ours actions. However, in other cases we might have a sense of these issues, but not act. We explore briefly why these user related mistakes are made but honestly likely to varies from situation to situation. Despite this variance common trends do emerge including:
Assumption that we are representative of typical end users broadly
Ex: WWW as Wealthy Western Web instead of World Wide Web
Users not caring how you built something (ex: What is a browser video?)
Not being mindful of who we are doing this for - ourselves or our users or our bosses
This appears to be driven by a few issues
Mistakes or assumption of our enviro = user enviro
Desire to satisfy our needs over our users - ex: Resume Driven Development
This often manifest in a tension between DX and UX (the developer - user see-saw)
Extreme time or cost pressures - economic constraints or flawed belief in first mover advantages
Flawed cultures we are encouraged to adopt - ‘the fish rots from the head problem’
This gets into our responsibility or not as developers - “Just a dev on the Death Star” problem
Premise - Process models for software aren't perfect, vary greatly, but are still very important
How to “play the game” of software development in a team fashion is not a physics or math based law. Given we have humans of varying skills and degrees of discipline working together we get highly variable outcomes, so it is unlikely we can find some single way of doing things. Giving up given such variability is unacceptable. We apply software engineering principles and processes as a way to increase the probability of a positive outcome (See Thinking in Bets). Such models have evolved over the years and it is my strong belief that we should know all the broad approaches and when to employ one or the other as opposed to rigidly embracing whatever is currently fashionable at our organization or within the broader industry.
Key Support Points and Ideas
Software Activities
The included activities in the lifecycle of building software do not vary and include:
Requirements
Design
Development
Validation (aka Testing)
Deployment
Operations
Various continuous activities in management also tend to be ongoing and lead to an evolutionary thought about software. In other words software grows like a garden as opposed to is built like a building.
Problem Solving Broadly
Top-Down Design - start with high level and break down towards small pieces. The common reductionist style
Bottom-Up Design - using interactions, data, etc. to grow upwards
Middle-Out - a surprising concept that has us trying things sometimes completely out of order and still finding value via exploration
Generally I think you need to move between all the techniques as appropriate.
An easy way to recall that realism of all approaches being useful might be to think of writing a book. Start with concept and outline - top-down. Write chapters and realize outline needs revision - bottom up. Get stuck and try sections or examples out of order - middle out
Incremental and At Once
Incremental solutions focus on doing pieces at a time
An all at once solution suggests we do the whole thing in one go and release
Generally incremental solutions provide more safety (check as we d, but truthfully a system often isn’t viable until many increments are done. Think a Jet Plane a tail, or cockpit might be an increment but we need many increments before we have something workable
Incrementalism tends to associate with iterative processes since we often do something release, get feedback and adjust.
Incremental approaches on the surface feel much safer because you can see progress and get feedback as opposed to doing something for a really long time and release all at once.
My general experience with this is somewhat varied. Start-up companies very much find this approach useful, but some types of software it doesn’t make much sense at it is too risky. (Think space probe software)
Metaphors from little to big don’t seem to hold up and often there are large scale changes. Recall the caterpillar to butterfly idea or skateboard to car evolution. Incrementalism is rarely as neat and tidy and risk free as many people advocate
Linear and Iterative
One problem solving approach is more of a step at a time to completion linearly approaching some done state.
This linear approach eventually can be boiled down to a check list, recipe, dare I say algorithm once we understand a problem well enough. The problem is that such problems can and should be automated.
More interesting problems probably can’t be linearly procedurized at least not at first so we may need to cycle over the problem many times. An iterative approach has us creating a little bit of the code at a time evolving it if you like.
Many people focus on the idea of evolving or growing software as opposed to assembling software.
I actually am fine with both as I think iteration is very much part of design engineering where a linear is very much a process of production engineering. Software contains both of these forms of engineering so I prefer more of a this AND that as opposed to this OR that.
The Iron Triangle
The Iron Triangle represents what most consider a fairly inflexible law of project management especially software project management. The graphic below shows a representation of the triangle with a sense one can pull on each of the handles
The idea here is that if you pull COST to be low it reduces quality. You pull scope it probably effects cost (higher) and schedule (longer) and may reduce quality.
For me I think the key points is trying to fix from 1 - 3 of these so you can then accurately focus on quality. If cost is set then it limits scope and schedule to what can reasonably be done with a defined quality. If schedule is set same thing. If scope is set same thing. However, when you set more than 1 it gets us to a place where you might have fancy features (big scope), but low budge (too little cost), or no time left because of procrastination (poor schedule)
My best advice is not focus on the pull points, but focus on the acceptable quality. Without defining quality we are somewhat lost unfortunately.
TL;DR - fix one or more of these and assume that if you pull you give things up.
Trade-offs Part Infinity
The iron triangle reminds us that a key aspect of engineering is embracing trade-offs. We must admit we can’t have it all and we try to balance our requirements with our situation as best we can to make a “good enough” solution as opposed to some perfect solution.
Consider even if you can make a perfect solution, it likely will only be perfect under a particular context and for a particular period of time. If context changes open what is a “perfect” solution changes.
Common Trade-off tensions include:
performance vs security - high performance, lower security vs higher security vs lower performance
security vs usability / ux - more security tends to degrade ux and inversely ease of use might sacrifice security.
Fashion Based Problem Solving
Acknowledge that the market does effect what we do as SEs. It is a common topic in the course, but we see in the process model space consultants pitching classes and books to take their waterfall+ or agile’ twist and sell you on that it solves everything. This is not an unusual concept and likely anything sold has that to some degree, but why this is a point to consider with process models is that software projects are tough and risky so management can and does fall for fashionable solutions. When you encounter such things you need to work at how to describe and explore these fashionable solutions to see if there are novel or just trendy.
Process Models and Time
A key aspect of process models that management in particular cares about is being able to estimate when a projects will be finished. Management would want some certainty like what this shows. In a school environment that indeed might be something like a set deadline. Companies can do this as well. However, to meet that deadline we likely may need to sacrifice quality and features which then is likely reflected in our grade or industry lower user-acceptance or even profit.
The truth of estimating and completion when you don’t have a fixed date is that things tend to shift to hit the desired quality mark. This chart shows the more likely reality and your goal is to be closed to the dotted line with the quality required.
In SE It's Always People
There are many examples of people being the #1 force in Software Engineering from focusing on individual abilities, the role of the 10x developer, group dynamics, decision making, user centered thinking, requirements gathering, user acceptance testing, and more people are at the center of nearly every aspect of SE.
The big challenge with people is that they frankly introduce a form of variability that is simply quite difficult to account for. The certainty of the programming activity for example is made hard by the uncertainty of your fellow humans in the form of your stake holders (product owners), team members, or end users.
The fact that people dominate SE is not really that controversial and numerous books and works have validated this idea. However, we note in the discussion this truism because as we get more in process and eventually tools and code we want things to be more predictable and less fuzzy, but that sadly isn’t very true.
The Cone of Uncertainty
The cone of uncertainty is a useful way to remind ourselves that when starting something we are mostly uncertain and the longer time or examination continues especially as we build software the more certain and specific we get.
Thinking in Bets
The process models and estimation aspects of SE suggest a certain amount of prediction. Prediction of the future is of course not a science. There are various heuristics including the obvious observing past behavior to estimate future outcomes, but given the variability of the world this will not always work. A good way to address this is to employ a thinking in bets type strategy.
The basic idea of thinking in bets is instead of 1 / 0 certainty of pass or fail you adopt more a percentage outcome of pass fail like 86%/14% or something. Thinking of poker hands where you have a high chance to win but no guarantee is a good way to recall how to employ this thinking.
Spectrum of Process
The spectrum process runs from pre-planned very formal linear processes to ad hoc and near choatic processes like “hacking” or “cowboy coding”. Recall that DUF = Design Up Front and this just showing how much work we do before we start the development.
A key point of the spectrum is to understand that the need to employ one process or another might depend more on the risk appetite, project type, time availability, team maturity, organization needs, and such as opposed to one particular right or wrong way. The diagram here is an alternative representation of this spectrum with some region markers and examples to make more sense of the thought.
“Waterfall” / Linear Approaches
The idea of the waterfall as a process model is to break up a software development projects into a set of steps running through designing, planning, executing, verifying and maintaining. The are various versions of this model. Some have 5 steps, some 7, some 8, etc. The names also vary. This one is a simple one we can use for understanding purposes
For testing purposes I’ll rely on on my own variations which adds a few modifications to the standard waterfall thinking. First, it breaks up the design process into more steps. Second it interlaces things like testing into development in particular unit testing as it should come during that process as opposed to after, and finally it shows arrows up suggesting a return to previous steps if warranted.
I generally prefer a modified waterfall where I have added a design whirlpool effort before entering the build process. I have found this a good way to launch projects with less risk in practice. Once done though we tend to fall into a spiral or agile style for further iteration work.
“Agile” / Iterative Approaches
Waterfall approaches do not handle change well, often do not get user feedback early and often enough without modification (ex. whirlpool idea), and frankly may delay progress or gratification in a world that demands to see that. Agile approaches such as Kanban, XP, etc. attempt to change this with more of an iterative approach which focuses on breaking the project into mini work efforts typically called sprints and growing your software incrementally with constant feedback.
Like waterfall/linear approaches there are also many versions of the iterative approach as shown in this graphic.
The Spiral Model for Boehm is one of the earliest well known iterative approach. Note that there are the waterfall style steps (recall these are just the SE activities) performed over and over again.
Depending on your degree of formality this ranges from very systematic to nearly ad hoc. Consider for example the “Sad JAD” (Joint Application Design) idea which one could say is iterative. This is the idea of getting vague requirements from a customer. Building and then having the customer react and repeating. This is Agile like but is incredibly inefficient and lacking in guardrails like estimates (story points), defined work units (sprints), user centeredness (user stories), and so on.
Our main focus here is simply to understand the range of iterative approaches which range from more linear/waterfall like to more ad hoc like. An up close look at Agile in our particular class varietal which attempts to reach some happy middle ground is in a separated deck and document.
Over, Under, and Pragmatism
A major point of the process model overview deck is to acknowledge there is a possibility of using an over engineered solution for a particular job (formal model for hello world) and under engineered or dangerous solution for another job (cowboy hacking or rapid iteration for a life and limb application like a heart monitor or autonomous driving system).
Developing a pragmatic attitude is a key aspect of a mature software engineer. It is unlikely we can come up with some one true way to build software for all situations. However, it is quite likely that thru experience and holding a pragmatic attitude you may find your way to adopting the appropriate solution to the team, situation and job you face.
If any one thing is learned about the process model it is that the how (process model) is only about supporting the creation of the what (software). If the what can be properly created meeting the organization constraints (cost, time, etc.) and quality goals than that is an appropriate solution.
SE and Process isn’t really special
Some final points about process models is that a group of humans creating something using a methodology is 100% not unique to software and you are encouraged to look at other group activities from a kitchen at a restaurant, an operating room, a flight deck, a war room, a soccer pitch, a movie production, a magazine or newspaper issue being made, a construction site, <insert dozens of others> for ideas. The idea that what we are doing is so utterly unique in my view is a bit arrogant and restrictive in terms of our growth as an industry. We can should learn from others.
Even if we stay insular to our space many ideas are long stated but not truly admitted to our adhered to, for example
“Knowing what to build is typically harder than how to build it” (Paraphrased)
“Requirements and ideas are not always clear to people and extracting them takes time and iteration” (Paraphrased and related to “Users/Customers aren’t designers”)
“Grow, don’t build software”
All these quotes are 50 years old and from titans of SE.
TL;DR - we’ve known many of the problems with process for a long time, but struggle as an industry to get consensus.
Conclusion
The point of a process model is to add organization to our problem solving in software engineering. However, problem solving is not unique to the development of software. We started first with understanding generalized thinking top down, bottom up, middle out as well as incremental and iterative concepts. Hopefully we have begun to adopt an engineering mindset of pragmatism. A mindset that accepts trade-offs and understands the iron triangle law. Our mindset focuses less on perfect eternal solutions and more on situational appropriate solutions. The concept of risk mitigation, appropriate model for situation and thinking in bets were different ways to express that type of situational pragmatism. Finally we should be very concerned about one true way and understand that from waterfall to agile to full on hacking we kind of need all of the approaches at our disposal. We must be quite careful to not believe that our form of problem solving is so utterly unique, because at the end of the day it is just a bunch of humans working to solve problems. We SEs do need to find both what is different with software, but also understand what is the same otherwise we miss easy improvements in our process of building things people actually want.
Epilogue - Process Models are Responses
Process models are really more responses to conditions, at least they should be, than things that live outside of any context. Waterfall was a product of how software was made, who funded it, and its costs. (Bigger corporate, government, etc.). Agile arose out a different period that favored a “first mover” idea and lack the previously required formality. Clearly Agile wasn’t dreamed up in some Platonic perfect way is to deny the dotCom context. Context is important and its is likely as conditions change via market, regulation, social expectations, the rise of AI, etc. process models will likely respond and react.
TL;DR - these are often lagging not leading ideas.
Figuring out what to build is the goal of the brainstorm and design engineering process. Figuring out the specifics and defining them is the point of requirements (what exactly the software should do) and the specifications (how we might roughly go about it organizationally). Architecture which follows gets even more precise on the how getting to the system and code level. Our goal with reqs and specs is a bit more high level, but we see that very quickly we can forget about users, become impatient, avoid important details and start relying more and more on assumptions.
Key Support and Ideas
Planning is Cheaper and Faster than Coding
A simple sketch of a UI or an algorithmic outline is easier to revise and work out issues than laying out code.
People may resist planning because of impatience, time pressures, and it is not likely as rewarding as writing code (this may include the assumption you aren’t making progress by bosses)
Tip - Do you planning in comments or the Repo somehow?
Gathering Requirements is not unique to SE
The Who, What, When, Where, Why and How are the same things we need for requirements for software as well as for constructing something, baking something, writing a story, solving a crime, etc.
Ask the 5Ws and 1H or whatever other common approach to properly focus on what is needed. If you just focus on the H part (How) particular our approach you’ve under thought it and are too solution focused as opposed to user focused.
Someone has to define the requirements
If you skip requirements that doesn’t mean there aren’t any just that they aren’t defined in an a transparent manner. In other words the requirements might live in some boss, stake holder, or customer head only to be revealed when your software meets or fails them might you start to get a clue of those requirements
Given the risk involved SEs should be involved in the creation of requirements or at least the verification of the requirements
Getting Requirements from People is Hard
We must translate from people often unskilled in software to the precision we need to build software
The requirements are more for us than to be clear than as some transcription of exactly what they said.
It is interesting to note that the lack of precision in user provided requirements and vague language use and how customers may then be dissatisfied with our software because we didn’t understand what they meant isn’t that unlike prompting an LLM system with a vague prompt and being surprised that it didn’t produce the exact code we wanted. Maybe by using these systems we can gain more empathy of users and also be more precise in our language use for problem description.
Quantitative and Qualitative Requirements
The truth is you’ll have both in your project such as “it needs to look clean and modern” and “the app must load in < 1second”
Obviously it is much easier to measure our progress towards precise quantitative requirements and you should have many to guide your efforts, but accept that some requirements will be more qualitative.
When faced with qualitative requirements try to add some quantitative aspect to steer. Ex: High Security → No major vulnerabilities discovered by a scan with tool X. Without this mapping you run the risk of some assumption of the meaning of the qualitative statement. Adding a metric will help you uncover some shared yardstick
Application: Make sure you can identify poor requirements and write or rewrite requirements into more concrete and hopefully quantifiable requirements.
Find Your Required Formality
Depending on your projects time and risk there is a range of formalism you might adopt from a few simple documents and a set of user stories to structured templates
There will be no single best way to document your system’s requirements, so figure out what culturally is required in the organization you find yourself in. Ex: there might be a vast difference between the highly structured docs before coding in a DoD project than a start-up which may lack much of anything.
Don’t assume one degree of formality is right and one is wrong, just that one is more appropriate than another in a given context with different practitioners.
The most important thing about documentation is often less the particular approach but whether the formality is appropriate and the approach applied consistently.
Visualizing Requirements
Visuals often communicate more efficiently than words, your requirements and design documents will likely have both things
Diagrams are commonly used to describe system architecture and flow
Common diagram types include:
Flowcharts
UML Diagrams
C4 Diagrams
Sadly mostly we find less commonly employed diagrams and more Ad Hoc Boxes and Arrows which is better than nothing, but our industry might benefit from shared blueprinting thought
Diagrams are also commonly found in interface specification
Wireframes
Interface Flow Diagrams and Clickable Prototypes
Be very cautious with making your interface diagrams too high fidelity otherwise people may over focus on colors or style choices rather than key requirements and decisions
Tip: Keep these things in your repo or very near your repo (project wiki)
Focus on Users Over Features
As developers we often think in code and features, this is of course not how users think
User tend to think in needs, wants or as some like to say “jobs to be done”
One way to think of jobs to be done is “I use <think> because it helps me to accomplish <outcome>”
Notice how we have just defined Agile’s user stories here! Write user stories and use cases and always think of real users in mind as we define our software.
Notice how the jobs to be done is somewhat more focused on a user mental model discussed next.
System Model vs User Model
The system model is how we tend to think of our code and present it it is in a sense our mental model as devs.
System models tends to focus on data and feature thinking and is helpful to organize code, but unhelpful to users how to accomplish their “jobs to be done”
A user model is how the user thinks of a task/product and is often more function based and less feature or software based.
Spolsky’s cardinal Axiom suggests that we focus on User Model > System Model
An example was given with a Web Application Firewall (WAF) called SeverDefender that employed a system model and failed at first despite being technical superior. Applying a user model to the situation change that outcome and was a tremendous simplification interface wise (though not technology wise). A key point from the story should be that the architecture of the system allowed for us to rectify the system model - user model gap, very often that isn’t possible so I generally strongly encourage you to work user model (often interface) downwards. Do note I was attempted to follow user model, I just listened to domain experts and non-representative users too much :-(
Have a SSoT (Single Source of Truth) for your design and requirements
Admit that your software is much greater than just code (Software >>> Code)
The information that is important in software includes our plans, diagrams, user stories, meeting notes, internal docs, external docs, test cases, etc.
If we keep all these things spread around we are unlikely to keep track of everything well and may lose data over time. The loss of data about the software and code can be devastating to the long term success of the software. If the info spreads or is lost or fuzzy because it is only in people’s heads the software is in effect “dying”.
A suggestion in this course is to keep everything in the repo or repo + wiki as our single source of truth (SSoT)
Ponder the Possibility of Strict Reqs and Specs for Systems
If we make our specifications and requirements very very precise we might see a possibility of generating code from such things.
Formal definition though is harder than it looks and might be difficult to do especially in a world that wants to change things all the time
At this current point in time (2023) the use of ML based prompt systems should show a possibility, but also reveal some of the downsides as well! These systems are looser especially the way we prompt them than a formal specification translated to code.
Conclusion
It is far easier to work out our thoughts before we start building. The reqs and specs should be defined and the architecture eventually hammered out as best we can. We must be careful not to over do it lest we lose agility, but we must do some of these activities before we start otherwise we add massive risk to our projects.
The core idea of Agile is that we must as developers display agility - an ability to move quickly and easily - given that we are likely facing constant change and discovering things as we build our systems. The Agile movement born over 20 years go now has become the de facto way a significant amount of software is built, so understand its values, ceremonies, benefits and problems is key to succeeding as a software developer today. A warning is given before continuing that like the idea of SE itself the particulars of what is thought to be Agile will vary from organization to organization and in some cases its application will be mostly in name only.
Key Support Points and Ideas
Which came first Agile or the need?
In my opinion Agile is a response to the failures of traditional SE models during the dotCom period.
Like most things in software engineering it is lagging indicator and reactive and supportive to conditions felt by businesses that fund us.
They historical baggage of Agile should be admitted to as well as the change of context.
TL;DR - Agile came after its need, not before like most things in SE
Digging Into Agile Values
We can look at the values in the Agile manifesto, but the key values from Extreme Programming (XP) are a better short hand and include a focus on
Communication
Feedback
Simplicity
Courage
Notice the focus on (soft) skills here we have already covered.
The specifics from the manifesto are
Individuals and Interactions OVER Process and Tools
Working Software OVER Comprehensive Documentation
Customer Collaboration OVER Contract Negotiation
Responding to Change OVER Following a Plan
The Agile Manifesto
The manifesto has 12 pieces but you should be aware of the main points of it and I will bold the key words that are the essence of it
Customer Satisfaction by rapid delivery of software
Welcoming changing requirements even late in development
Working software is delivered frequently (weeks not months)
Working software is the principal measure of progress
Sustainable development, able to maintain a constant pace
Close, daily cooperation between business people and developers
Face-to-face conversation is the best form of communication (co-location)
Projects are built around motivated individuals, who should be trusted
Continuous attention to technical excellence and good design
Simplicity - the art of maximizing the amount of work not done - is essential
Self-organizing teams
Regular adaptation to changing circumstances In short at it's heart Agile is about speed, user focus, communication, self organization, good tech and design, keeping it simple and dealing with change.
Agile Ceremony: Daily Stand-Up
Quick meetings of what you did, what you want/need to do, and any blockers to be cleared
The deeper point is sharing what we are doing as a group, focusing on alignment, setting priorities, and making sure we have nothing impeding our progress. It insures we are on the same page with a shared goal and understanding of work to be done and capable of doing said work
The usually are done synchronously and in-person and each speaker should finish in a few minutes.
Tip: Things that take longer should be taken offline for other meetings or outside follow-up
The idea of the stand-up is absolutely not novel - think of a football huddle, the pre-operation briefing before a day in many jobs, etc. I had stumbled upon this early in my career and used an egg timer to run my morning meeting. The ticking really helped people get to the point!
Tip: Set time, place, structure and time blocking really helps this ceremony to be useful.
Agile Concept and Artifact: (User) Stories
As we discussed in the UCD lecture user stories help us write user needs in a manner that we might translate them into the requirements for our project
The format is “As a [user who is focus of story] I want [to accomplish the goal of the story] so that [reason for the story]”
Example: As a student of CSE 110, I want clarity of information, so that I can more easily score a good mark in the course.
Translation of the user stories often includes a format for acceptance criteria “Given <some context> When <some action is carried out> Then <a set of observable outcome”
Example: Given they are willing to adequately study for the midterm when they consult the new slides then the understanding gained there will be clear enough they ace the midterm”
That silly example might be better as features (improved slides), so you can see the gap from stories to execution still leaving us plenty of room for executional details.
Agile Concept and Artifact: Backlog
A backlog in Agile is a collection of stories that have been defined for the product.
Developers, product managers, or both often develop these stories and put this in a tool (ex. Jira, Github Issues, etc.) or even on 3x5” cards or post-its
In my experience a backlog that is too large is a sign that you are actually doing a form of disguised waterfall just without sequencing built-in.
TL;DR - backlogs contain what you will eventually do, but you should aim to limit your WIP (work in progress) by keeping them small and being more detailed with things to come soon and less for things way out (relates to the cone of uncertainty!)
Agile Concept: Sprint
Under Agile a sprint is defined as a time box to conduct work under.
Sprint lengths are more often two weeks, but you will see variation from 1 to even 6 weeks
The idea of a time-box is to create a particular cadence and segmentation of work
Some of the ideas of sprints are a form of mental chunking that is helpful for people to sustain themselves over time.
WARNING: The idea of a sprint implies speed and in some cases the velocity and approach of this methodology particularly in the hands of organizations that like to “whip the ponies” is what leads to burnout and job jumping.
Agile Ceremony: Sprint Planning
At the start of the spring a longer meeting called the Sprint Planning meeting is held where developers get together and pick items out of the backlog to work on
Knowing what and how many stories can be fit into a particular sprint is a bit of an art, but involves of prioritizing things (what - which fits with the degree of importance or biz goals) and how many fitting with how big each task is using a form of virtual work currency (story points)
Given that we may need to agree as a team what to commit to and how long each thing might take an activity called Planning Poker is sometimes used. My experience with this is that it is relatively rare and of questionable value unless all team members understand the code base equally which is super unlikely
Agile Concept: Story Points
The idea of the story point is to remove the assumed precision of estimates as it is really hard to know exactly how long something would take. (2hrs vs 8hrs)
Instead of just giving up the Agile creator included the idea of story points similar to say t-shirt sizes like s, m, lg, xl.
The size buckets allow us to suggest a range of get a sense of rough degree of time and difficulty involved so we can prioritize our time like some form of task Tetris
Some organizations try to get this formal by creating a story point budget and then over sprints getting better and better at knowing what the point capacity is for the team
Agile Artifact: Burn Down Chart
Managers often want to know how close are we to done so in Agile we often will find a burn down chart which tracks the allocated values of stories (in story points) showing how much we have completed (our progress) and how much is left to go.
When done well these charts can give us a good sense if we are planning too much or too little for sprints and help managers and the team know if we are on target for finishing the work we promised
When done poorly it is a sign of a command and control organization that is not really empowering devs despite what Agile may encourage them to do and could result in managers jamming more hours in to make sure the sprint gets everything done as opposed to missing and adjusting our capacity to some reasonable amount.
Agile Ceremony: Sprint Review
The end of the sprint has a meeting which is a form of show and tell of what the sprint accomplished.
The meeting also allows us to see what wasn’t accomplished and move back into a backlog
This meeting relates to the retrospective which talks about the process of how we played and sometimes the meetings are intermixed.
My experience we often do mini “show and tells” as we go and make the weekly or sprint end one much longer and in-depth
Agile Ceremony: Retrospective
A retrospective meeting allows us to look back at our previous sprint and discuss the high level issues of Agile and what went right and wrong
Retros should be run without fail as a team is built and when really bad or good things happen especially if surprising. The goal is to understand how we might do better the next go around in the case of bad or understand why things went so well.
Retrospectives are often missing in organizations as they do not value the introspection as non-billable work so to speak. This is an unfortunate oversight and that is why I modify it here to the new team looking to rhythm and to react to overly positive or negative situations as a reasonable response to the push back from organizations
Agile Always Varies
Many of the previous segments have had discussion of variations, reductions or even outright statements that the ideas are more aspirational than actual in most cases. A take-away from that is like many things in SE it isn’t one exact thing, but an essence of something.
Assume that you need to get the “flavor of Agile play” at the org you join. Listen to their “rules” observe the ceremonies and be careful not to immediately assume the way you did it elsewhere including school makes sense at the current place.
TL;DR - The specifics of Agile likely always varies, but the intent shouldn’t
Agile Fails
Agile does not solve the difficult of software engineering, but it can help since its nature is to admit we have to figure things out as we go to some degree and be flexible with what is thrown at us
Inherently school teaches us waterfall with a spec in a way so it isn’t surprising that what I teach and how I teach it shocks you!
Sadly Agile fails way too often in industry from a variety of reasons including:
No doing any planning up front because of Agile
Working too hard because the emphasis on velocity and doing poor work or burning people out
Not truly empowering devs dictating what they do into some feature factory grind with a typical hierarchical structure but telling them it is really Agile. This is more common than you think even within FAANG so watch out for it.
Pushing things out prematurely with user impacts
Creating its own Dogma just as stodgy as the worst waterfall shop especially at scale
Profs Agile Mods
Allowing for the Andon Cord
Sprint length variability
Sprint type variation & mental crop rotation
Dumping the story points for honesty that we always translate anyway
Conclusion
Agile is based on real things and has set ideas, but the truth is few places really live up to all the aspects of Agile, but hopefully live up to its intentions. We should first embrace our form of the basic values of Agile and its practical goals. That is the most important thing rather the specific form or adherence of things like stand-ups, burn-downs, retros, and the like. In short the point is delivering software and not subscribing to the dogma. The creators of Agile railed about such waterfall dogma and as the epilogue says at least at scale we might be right back there waiting for whatever your generation brands as the new (mostly the same) way since all are just the rough process ideas we mentioned in the general discussion underneath.
Epilogue - The Emperor Has No Agility?
Agile has been promoted as the savior of all things, but over 20 years it clearly has not done that. While much software has shipped there has been excessive waste and constant failures. We even see a rise of an Agile Industrial Complex filled with coaches, certification, methodologies and all manners of bureaucracy that would make the waterfallists of the 70s to the 90s feel very much at home. Many large organizations are actually pretty poor at software engineering in that they are slow at it, reckless (moving fast and breaking things including societies), or quite inefficient requiring armies of devs to do basic things or replicating things all over their org because nobody knows what anyone else is doing because of aggressive autonomy. I’ll leave this solely as a warning which some students will also share from internships that you might find people say they are doing Agile more than they are actually being Agile even at some of the firms you very much aspire to get into. Remember never equate a company brand or revenue with SE maturity, those are two separate things. Many of the Emperors have no Agile clothes even if they believe they do and claim their pay rate or market position says it is so. This is no different than the big company ideas of the 80s, 90s, and early 00s just with much better perks!
TL;DR - Don’t be surprised if you are disappointed with what you see in industry if you believe in the ideas of Agile, have the courage to try to improve and the ability to laugh at the absurdity of it when you can’t.
Architecture is an important activity that is moving missing or done as we go along in Agile environments. There is a tension though about overdoing it with architecture as well, so like most aspects of engineering we need to find a balance.
We explore a number of common architectural patterns such as layered, microservice, monolith, MVC and more. However, I am concerned that whether at the architectural or code level a pattern can be a crutch that makes things worse. We often fit the problem to the model as opposed to find the model that fits the problem.
Since architecture influences everything and tends to include decisions that are hard to undo, we should take enough and certainly document our choices. A common artifact you should become familiar with is the Architectural Decision Record (ADR) which serves as a reminder of the thoughts that went to making a particular important decision. Diagrams are also key artifacts for architecture and should be kept up to date and provide the appropriate granularity from high to low level.
Key Support and Ideas
Architecture is Very Important
Architecture as Martin Fowler says covers the important stuff and the hard to change stuff, if you believe that raw definition you really need to get it right then!
Architecture effects things both users and developer feel. We often describe those things as the “-ilities” such as
Availability
Maintainability
Usability
Security
Performance
Reliability
Real and Fake World Architecture Examples
I provided two examples of architecture success and failure in the case of the Sydney Opera House (a failure that becomes a success) and the Darth Star (a success that becomes a failure)
The only point of those examples was to show the outcome style thinking here. Both examples had huge mistakes architecture wise, one recovered albeit with great cost and trouble to become a cultural icon. The other was massive powerful and got taken down by a tiny space ship. Both things exist out in our tech world, be very cautious of survivorship bias that allows you to waive away architectural impacts. (ex: look at poorly built system that made people a boat load of $, clearly that was the risky route!)
Architect Absent or All Powerful?
Agile tends to focus on less global roles so architectural duties are not explicitly defined. In some ways, it would be thought to be emergent from the group with individual members making decisions. This lead to my point of Where’s Waldo the Architect? for Agile.
The architectural guru, astronaut, “all powerful”, etc. is at the opposite end of the spectrum. The challenge here is that such a person is hard to find, likely becomes a single point of failure and frankly becomes too distant to make good decisions.
My general thought is we need an architect and they should be experienced, but trust their hands on members if not be in the code with them!
Getting It Done vs Getting it Right
In our market driven world, so called scrap heap programming or copy-paste coding if you like does result in the “GETS” (good enough to ship). However, just getting it done generally comes with a long term cost.
Tech Debt and Tech Investment
The idea of tech debt is that the short term decisions of the last section will tend to add up over time. The interest on this tech debt can become large as we need to fix bugs or the velocity of making changes slows down as the system becomes the so called big ball of mud
Tech debt isn't inherently bad in some ways in that as we previously mentioned it might be ok for expedience to take some on to get a short term gain, of course that assumes you pay it off.
If we are going to use the idea of tech debt, then the idea of tech investment where we lay down tech, do research, educate ourselves, clean a code base, etc. should be expressed as well.
Balance Your Tech Decisions
Perfect tech decisions simply don’t exist, instead you just aim to make the best decision you can at a particular time with a particular awareness of concerns. If those things change the decision may not be good any more. In short a perfect decision now, might not be one later.
Balancing tech decisions usually boils down to solving the problem in front of you with awareness of potential problems in the future.
I dub the Lambo Problem idea as a way to avoid making inappropriate decisions like scale, automation, generalization, etc. early. In short worry about the Lambo color you might choose when you get that successful.
Broad Architectural Catch Phrases
These catch phrases and concepts are good, but you need to apply them and avoid slavish adherence because they tend to have have counter-points. I’ll put those in the commentary
YAGNI - You Ain't Gonna Need It - keep the features load, the abstraction down, the tools simple, etc. because likely you won't end up needing them
KISS - Keep it Super Simple, Keep It Simple Stupid, etc. - like the last phrase, but a focus mostly on complexity
DRY - Don’t Repeat Yourself - like a single source of truth for code or an idea, duplication seems to be a bad idea since you need to find and update all occurrences and it just makes a system bigger
Form Follows Function - let the system follow what is defined to do, usually is related to interface
Accept Trade-offs - usually you can’t have everything, security might mean lower performance, higher security might lower usability, etc.
Broad Concept - Decomposition
A key job is breaking our code and systems into the most appropriate sized parts.
If a system is large we tend to dub it monolithic. If it is broken apart we used to dub it decomposed, now we tend to label it as a microservices. The general idea of microservices is to provide a single responsibility for things. This follows the idea of the Single Responsibility Principle (SRP) which should lead to less complexity to reason about and test.
Like most things decomposition isn’t something that you do and are finished with, it tends to require revisiting over time. For example, you might start out with a monolithic module and start breaking it down as it grows. Be very careful overly decomposing at first as that is the premature abstraction we discussed in the previous section.
A layer pattern also is a common decomposition with things below providing services for things above. A common layer pattern we see is the Model View Controller (MVC) pattern.
Broad Concept - Coupling
Coupling is the idea of related things together. If component A does not relate or depend on B it is not coupled, If component A and B relate only we have a loose coupling. If a component depends on many components it is highly coupled. Generally we want loose coupling as it is easier to swap thing out and to reason about them.
Like decomposition it is a balancing act, sometimes over decomposition leads to more coupling and communication overhead!
Broad Concept - Communication
Decomposed objects that are related do need to communicate. We alluded to that with coupling.
The overhead and failure possibility of communication should not be ignored, especially in a networked environment.
Communication patterns may be synchronous or asynchronous and communication events can be a good indicator of how a system actually is driven.
Event driven thinking is quite important in network systems and it is my opinion that it models systems with humans a bit better than synchronous algorithmic thinking.
Broad Concept - Architecture and the March of Time
Even if you make really good decisions over time systems will need to be changed or they will tend to fall to ruin. This shouldn’t be surprising if you consider software lifecycle or the simple idea of “technical thermodynamics” (a trend towards entropy)
ADRs
If we make an architectural decision we must admit that that decision is quite important and potentially long lasting or permenant.
The idea of an Architectural Decision Record (ADR) is that we capture our important decisions in easy to access and consume documents for ourselves and future developers.
For the course we recommend the Markdown or MADR format of ADRs.
Conclusion
Architectural activities should be performed in a software project regardless of the process model adopted (agile, waterfall, hybrid, etc). In short these activities happen whether you are mindful of them or not. Making these decisions in haste or without thought can result in catastrophes, though to be fair we can get lucky. Do not let survivorship bias encourage you to think you stumble upon architecture as you build.
Key phrases like KISS, YAGNI, DRY, etc. are good ideas, but code wise we should focus also on appropriate decomposition, coupling and communication. Over time what is thought a good decision on those thoughts may change so the architecture hopefully can evolve. Time will wear systems down so we need to plan to improve and capture important decisions for easy access using diagrams and ADRs.
