This reviews how different guidelines / success criteria can be audited, as part of sustyweb#29: Manual VS Automated Testing.
There are three tables to compare:
Last active
December 12, 2023 13:16
-
-
Save thibaudcolas/589de9192a919e41b67eefc98e0f4dac to your computer and use it in GitHub Desktop.
WSG 1.0 Manual VS Automated Testing comparison tables
Guideline-level table, started by @marvil07-adapt, expanded with audit type classification from @thibaudcolas.
Conversion from @thibaudcolas ratings to @marvil07-adapt:
- Static analysis → yes
- Automated → yes
- Manual, quantitative → partial
- Manual, qualitative → no
- Consulting → no – internal
Note for @thibaudcolas automation was only considered possible if the information was publicly available. The potential for internal automation wasn’t considered, except for static analysis (where internal information would help but isn’t a requirement)
Conversion from SC-level rating to guideline-level: use "partial" if the SCs would differ in the yes/no scoring.
How we match up across 93 guidelines:
- Same assessment for 69 guidelines
- Almost the same for 21 ("partial" vs. yes or no) – indicated as "🚧 Diff"
- Different for 3 ("yes" vs "no") – indicated with "🔴 Diff"
| @marvil07-adapt | @thibaudcolas | Guideline |
|---|---|---|
| no – internal | no – internal | 2.1 Undertake Systemic Impacts Mapping |
| no – internal | no – internal | 2.2 Assess And Research Visitor Needs |
| no – internal | no – internal | 2.3 Research Non-visitor's Needs |
| no – internal | no – internal | 2.4 Consider Sustainability In Early Ideation |
| no – internal | no – internal | 2.5 Account For Stakeholder Issues |
| partial | partial | 2.6 Create a Frictionless Lightweight Experience By Default |
| partial – editorial | 🚧 Diff: no | 2.7 Avoid Unnecessary Or An Overabundance Of Assets |
| partial | 🚧 Diff: no | 2.8 Ensure Navigation And Way-finding Is Well-structured |
| partial | 🚧 Diff: no – internal | 2.9 Respect The Visitor's Attention |
| no | no | 2.10 Use Recognized Design Patterns |
| partial | 🚧 Diff: no | 2.11 Avoid Manipulative Patterns |
| no – internal | no – internal | 2.12 Document And Share Project Outputs |
| no – internal | no – internal | 2.13 Use A Design System To Prioritize Interface Consistency |
| partial | partial | 2.14 Write With Purpose, In An Accessible, Easy To Understand Format |
| partial – editorial | partial | 2.15 Take a More Sustainable Approach To Image Assets |
| partial – editorial | partial | 2.16 Take a More Sustainable Approach To Media Assets |
| no – editorial | no | 2.17 Take a More Sustainable Approach To Animation |
| yes | yes | 2.18 Take a More Sustainable Approach To Typefaces |
| partial | partial | 2.19 Provide Suitable Alternatives To Web Assets |
| no | no | 2.20 Provide Accessible, Usable, Minimal Web Forms |
| partial | 🚧 Diff: no | 2.21 Support Non-Graphic Ways To Interact With Content |
| no | no | 2.22 Give Useful Notifications To Improve The Visitor's Journey |
| partial | partial | 2.23 Reduce The Impact Of Downloadable Or Physical Documents |
| no – internal | no – internal | 2.24 Create A Stakeholder-focused Testing & Prototyping Policy |
| no – internal | no – internal | 2.25 Conduct Regular Audits, Regression, And Non-regression Tests |
| no – internal | no – internal | 2.26 Analyze The Performance Of The Visitor Journey |
| no – internal | no – internal | 2.27 Incorporate Value Testing Into Each Major Release-cycle |
| no – internal | no – internal | 2.28 Incorporate Usability Testing Into Each Minor Release-cycle |
| partial | partial | 2.29 Incorporate Compatibility Testing Into Each Release-cycle |
| no – editorial | no - internal | 3.1 Identify Relevant Technical Indicators |
| yes | yes | 3.2 Minify Your HTML, CSS, And JavaScript |
| partial – internal | 🚧 Diff: yes | 3.3 Use Code-splitting Within Projects |
| no – internal | 🔴 Diff: yes | 3.4 Apply Tree Shaking To Code |
| partial | partial | 3.5 Ensure Your Solutions Are Accessible |
| no – internal | no – internal | 3.6 Avoid Code Duplication |
| no | no – internal | 3.7 Rigorously Assess Third-party Services |
| partial | 🚧 Diff: yes | 3.8 Use HTML Elements Correctly |
| yes – editorial | 🚧 Diff: partial | 3.9 Resolve Render Blocking Content |
| partial – semantics | partial | 3.10 Provide Code-based Way-finding Mechanisms |
| partial | partial | 3.11 Validate Form Errors And External Input |
| partial – editorial, semantics | 🚧 Diff: yes | 3.12 Use Metadata Correctly |
| partial – editorial | 🚧 Diff: yes | 3.13 Adapt to User Preferences |
| partial | 🚧 Diff: no | 3.14 Develop A Mobile-first Layout |
| no | no – internal | 3.15 Use Beneficial JavaScript And Its API's |
| partial – internal | 🚧 Diff: yes | 3.16 Ensure Your Scripts Are Secure |
| no – internal | 🔴 Diff: yes | 3.17 Manage Dependencies Appropriately |
| yes | yes | 3.18 Include Files That Are Automatically Expected |
| yes | yes | 3.19 Use Plaintext Formats When Appropriate |
| no – internal | 🔴 Diff: yes | 3.20 Avoid Using Deprecated Or Proprietary Code |
| no | no – internal | 3.21 Align Technical Requirements With Sustainability Goals |
| yes – internal | yes – internal | 3.22 Use The Latest Stable Language Version |
| no – internal | no | 3.23 Take Advantage Of Native Features |
| no – internal | no | 3.24 Run Fewer, Simpler Queries As Possible |
| partial | partial | 4.1 Choose A Sustainable Hosting Provider |
| partial – editorial | 🚧 Diff: yes | 4.2 Optimize Browser Caching |
| yes | yes | 4.3 Compress Your Files |
| no – editorial | 🚧 Diff: partial | 4.4 Use Error Pages And Redirects Carefully |
| no – internal | no – internal | 4.5 Limit Usage Of Additional Environments |
| no – internal | no – internal | 4.6 Automate To Fit The Needs |
| no – editorial | 🚧 Diff: partial | 4.7 Maintain a Relevant Refresh Frequency |
| no – internal, editorial | no – internal | 4.8 Be Mindful Of Duplicate Data. |
| no – editorial | 🚧 Diff: partial | 4.9 Enable Asynchronous Processing And Communication |
| yes | 🚧 Diff: partial | 4.10 Use Edge Computing |
| no – internal | no – internal | 4.11 Use The Lowest Infrastructure Tier Meeting Business Requirements |
| no – internal, editorial | no – internal | 4.12 Store Data According To Visitor Needs |
| no – internal | no | 5.1 Have An Ethical And Sustainability Product Strategy |
| no – internal | no – internal | 5.2 Assign A Sustainability Representative |
| no – internal, internal | no – internal | 5.3 Raise Awareness And Inform |
| no – internal | no | 5.4 Communicate The Ecological Impact Of User Choices |
| partial – internal | 🚧 Diff: no – internal | 5.5 Estimate A Product Or Service's Environmental Impact |
| no – internal | no | 5.6 Define Clear Organizational Sustainability Goals And Metrics |
| no – internal | no – internal | 5.7 Verify Your Efforts Using Established Third-party Business Certifications |
| no – internal | no – internal | 5.8 Implement Sustainability Onboarding Guidelines |
| no – internal | no | 5.9 Support Mandatory Disclosures And Reporting |
| no – internal | no – internal | 5.10 Create One Or More Impact Business Models |
| no – internal | no – internal | 5.11 Follow A Product Management And Maintenance Strategy |
| no – internal | no – internal | 5.12 Implement Continuous Improvement Procedures |
| no – internal | no – internal | 5.13 Document Future Updates And Evolutions |
| no – internal | no – internal | 5.14 Establish If A Digital Product Or Service Is Necessary |
| no – internal | no – internal | 5.15 Determine The Functional Unit |
| no – internal | no – internal | 5.16 Create A Supplier Standards Of Practice |
| no – internal | no – internal | 5.17 Share Economic Benefits |
| no – internal | no – internal | 5.18 Share Decision-making Power With Appropriate Stakeholders |
| no – internal | 🚧 Diff: partial | 5.19 Use Justice, Equity, Diversity, Inclusion (JEDI) Practices |
| no – internal | no | 5.20 Promote Responsible Data Practices |
| no – internal | no – internal | 5.21 Implement Appropriate Data Management Procedures |
| no – internal | 🚧 Diff: partial | 5.22 Promote Responsible Emerging Technology Practices |
| no – internal | no – internal | 5.23 Include Responsible Financial Policies |
| no – internal | no – internal | 5.24 Include Organizational Philanthropy Policies |
| no – internal | no – internal | 5.25 Plan For A Digital Product Or Service's Care And End-Of-Life |
| no – internal | no – internal | 5.26 Include E-waste, Right-to-repair, And Recycling Policies |
| partial – internal | 🚧 Diff: no – internal | 5.27 Define Performance And Environmental Budgets |
| no – internal | no – internal | 5.28 Use Open Source Tools |
Expected audit types for all success criteria, from the perspective of a consultant WSG specialist, with an audit methodology similar to WCAG-EM.
Types of audits classified from most to least automate-able across 232 SCs:
- Static analysis (6): Potential to write automated code checks that would run in CI / developer IDEs. Example:
jsx-a11y/alt-text - Automated (35): "Runtime" analysis – potential to inspect the product with automated browsing or equivalent and detect issues. Example: Axe
- Manual, quantitative (16): Likely manual auditing but with potential to follow a set scoring algorithm. Possible to create semi-automated tools to help with auditing. Example: Tab stops testing
- Manual, qualitative (50): Manual auditing with an element of interpretation. Can be done with publicly available information, reproducibility of findings a possible concern. Example: WCAG SC 3.2.4 Consistent Identification
- Consulting: Manual auditing requiring interpretation and internal knowledge of the project / organisation. Cannot be audited without behind-the-scenes access.
| Audit type | Success Criterion |
|---|---|
| 2. User-Experience Design | |
| 2.1 Undertake Systemic Impacts Mapping | |
| consulting | External Variables |
| 2.2 Assess And Research Visitor Needs | |
| consulting | Identify And Define |
| consulting | Visitor Constraints |
| consulting | Barriers And Access |
| consulting | Barrier Removal |
| 2.3 Research Non-visitors Needs | |
| consulting | Non-Human Impact |
| 2.4 Consider Sustainability In Early Ideation | |
| consulting | Wireframes And Prototypes |
| 2.5 Account For Stakeholder Issues | |
| consulting | Human-Centered Brainstorming |
| consulting | Ecological Brainstorming |
| 2.6 Create a Frictionless Lightweight Experience By Default | |
| consulting | Performance By Default |
| manual, quantitative | Efficient Paths |
| manual, qualitative | Patterns For Efficiency |
| manual, qualitative | Distraction-Free Design |
| manual, qualitative | Eliminate The Non-Essential |
| manual, quantitative | User-Initiated Actionable Content |
| 2.7 Avoid Unnecessary Or An Overabundance Of Assets | |
| manual, qualitative | Decorative Design |
| 2.8 Ensure Navigation And Way-finding Is Well-structured | |
| manual, qualitative | Navigation And Search |
| manual, qualitative | Navigable Sitemaps |
| manual, qualitative | New Content |
| 2.9 Respect The visitors Attention | |
| manual, qualitative | Respecting Attention |
| consulting | Avoid Distraction |
| manual, qualitative | Avoid Attention-keeping |
| 2.10 Use Recognized Design Patterns | |
| manual, qualitative | Design Patterns |
| 2.11 Avoid Manipulative Patterns | |
| manual, qualitative | Dark Patterns |
| manual, qualitative | Using Advertisements |
| manual, qualitative | Page Tracking |
| 2.12 Document And Share Project Outputs | |
| consulting | Deliverables Reusability |
| consulting | Deliverables Documentation |
| 2.13 Use A Design System To Prioritize Interface Consistency | |
| consulting | Design System |
| 2.14 Write With Purpose, In An Accessible, Easy To Understand Format | |
| manual, qualitative | Write Clearly |
| manual, qualitative | Content Formatting |
| static analysis | Search Engine Optimization (SEO) |
| 2.15 Take a More Sustainable Approach To Image Assets | |
| automated | Need For Images |
| automated | Optimize Images |
| static analysis | Lazy Loading |
| manual, qualitative | Sizing And Deactivation |
| consulting | Management And Usage |
| 2.16 Take a More Sustainable Approach To Media Assets | |
| automated | Need For Media |
| automated | Optimize Media |
| manual, qualitative | Labels And Choice |
| consulting | Management And Usage |
| 2.17 Take a More Sustainable Approach To Animation | |
| manual, qualitative | Need For Animation |
| manual, qualitative | Avoid Overburdening |
| manual, qualitative | Control Animation |
| 2.18 Take a More Sustainable Approach To Typefaces | |
| static analysis | Default Typefaces |
| static analysis | Font Optimization |
| 2.19 Provide Suitable Alternatives To Web Assets | |
| manual, quantitative | Open Formats |
| manual, quantitative | Font Subsetting |
| automated | Alternative Text |
| manual, quantitative | Audio Alternatives |
| manual, quantitative | Video Alternatives |
| 2.20 Provide Accessible, Usable, Minimal Web Forms | |
| manual, qualitative | Form Simplicity |
| manual, qualitative | Form Functionality |
| 2.21 Support Non-Graphic Ways To Interact With Content | |
| manual, qualitative | Alternative Interactions |
| 2.22 Give Useful Notifications To Improve The visitors Journey | |
| manual, qualitative | Notification Justification |
| manual, qualitative | Notification Control |
| manual, qualitative | Prompts And Responses |
| 2.23 Reduce The Impact Of Downloadable Or Physical Documents | |
| manual, quantitative | Printing Documents |
| manual, quantitative | Optimize Documents |
| manual, quantitative | Labels And Choice |
| 2.24 Create A Stakeholder-focused Testing & Prototyping Policy | |
| consulting | New Features And Perspectives |
| consulting | Resourcing And Viability |
| consulting | Training And Onboarding |
| consulting | Testing And Validation |
| 2.25 Conduct Regular Audits, Regression, And Non-regression Tests | |
| consulting | Regular Issue Testing |
| consulting | Non-Regression Tests |
| consulting | Regression Tests |
| 2.26 Analyze The Performance Of The Visitor Journey | |
| consulting | Measurement And Compliance |
| 2.27 Incorporate Value Testing Into Each Major Release-cycle | |
| consulting | Usage Changes |
| 2.28 Incorporate Usability Testing Into Each Minor Release-cycle | |
| consulting | Usability Testing |
| 2.29 Incorporate Compatibility Testing Into Each Release-cycle | |
| consulting | Compatibility Policy |
| consulting | Maintaining Compatibility |
| manual, quantitative | Frequent Testing |
| consulting | Mobile Friendly |
| consulting | Progressive Web Application's (PWA)'s |
| 3. Web Development | |
| 3.1 Identify Relevant Technical Indicators | |
| consulting | Performance Goals |
| 3.2 Minify Your HTML, CSS, And JavaScript | |
| automated | Minify Code |
| 3.3 Use Code-splitting Within Projects | |
| automated | Code Splitting |
| 3.4 Apply Tree Shaking To Code | |
| automated | Remove Redundancy |
| 3.5 Ensure Your Solutions Are Accessible | |
| automated | Accessibility Compliance |
| static analysis | Enhancing For Accessibility |
| consulting | Electronic Inequalities |
| manual, qualitative | Carbon Aware Design |
| 3.6 Avoid Code Duplication | |
| consulting | Remove Or Simplify |
| consulting | Iteration Over Recreation |
| consulting | Organize Code Arrangement |
| 3.7 Rigorously Assess Third-party Services | |
| consulting | Assess Third-parties |
| consulting | Third-party Implementation |
| consulting | Libraries And Frameworks |
| consulting | Self-Hosting |
| consulting | Avoiding Dependency |
| 3.8 Use HTML Elements Correctly | |
| automated | Semantic Code |
| static analysis | Avoid Non-standard Code |
| manual, qualitative | Custom Code |
| 3.9 Resolve Render Blocking Content | |
| automated | Asynchronous Code |
| manual, qualitative | Priority Loading |
| 3.10 Provide Code-based Way-finding Mechanisms | |
| automated | Metadata And Microdata |
| automated | Search Engines |
| manual, qualitative | Accessibility Aids |
| 3.11 Validate Form Errors And External Input | |
| manual, qualitative | Error Validation |
| manual, qualitative | Label Elements |
| automated | Allow Paste |
| 3.12 Use Metadata Correctly | |
| automated | Required Elements |
| automated | Meta Tags |
| automated | Structured Data |
| 3.13 Use CSS Preference And Media Queries | |
| automated | CSS Preference Queries |
| 3.14 Develop A Mobile-first Layout | |
| manual, qualitative | Mobile-First |
| manual, qualitative | Responsive Design |
| manual, qualitative | Alternative Browsing |
| 3.15 Use Beneficial JavaScript And Its API's | |
| consulting | Beneficial JavaScript |
| manual, qualitative | API Requests |
| 3.16 Ensure Your Scripts Are Secure | |
| automated | Script Security |
| 3.17 Manage Dependencies Appropriately | |
| automated | Dependency Management |
| automated | Dependency Necessity |
| automated | Dependency Updates |
| 3.18 Include Files That Are Automatically Expected | |
| automated | Expected File Formats |
| 3.19 Use Plaintext Formats When Appropriate | |
| automated | Beneficial File Formats |
| 3.20 Avoid Using Deprecated Or Proprietary Code | |
| automated | Deprecated Code |
| automated | Outdated Code |
| 3.21 Align Technical Requirements With Sustainability Goals | |
| consulting | Identify Requirements |
| consulting | Optimized Methodology |
| consulting | Expandability Considerations |
| consulting | Interface Impact |
| 3.22 Use The Latest Stable Language Version | |
| automated | Versioning |
| 3.23 Take Advantage Of Native Features | |
| manual, qualitative | Native Over Custom |
| 3.24 Run Fewer, Simpler Queries As Possible | |
| manual, qualitative | Database Queries |
| 4. Hosting, Infrastructure And Systems | |
| 4.1 Choose A Sustainable Hosting Provider | |
| consulting | Monitor Metrics |
| consulting | Equipment Longevity |
| consulting | Recycling Waste |
| automated | Renewable Electricity |
| consulting | Remaining Emissions |
| 4.2 Optimize Browser Caching | |
| automated | Server-side Caching |
| automated | Offline Access |
| 4.3 Compress Your Files | |
| automated | Server-side Compression |
| automated | Media Compression |
| 4.4 Use Error Pages And Redirects Carefully | |
| manual, quantitative | Error Pages |
| manual, qualitative | Redirection |
| 4.5 Limit Usage Of Additional Environments | |
| consulting | Unused Environments |
| 4.6 Automate To Fit The Needs | |
| consulting | Automate Tasks |
| consulting | Necessitate Tasks |
| consulting | Automated Scaling |
| consulting | Security Tooling |
| 4.7 Frequency For Refresh Is Relevant To Visitor Needs | |
| manual, quantitative | Refresh Frequency |
| 4.8 Be Mindful Of Duplicate Data. | |
| consulting | Data Backups |
| 4.9 Enable Asynchronous Processing And Communication | |
| consulting | Batch Processing |
| automated | Protocol Usage |
| 4.10 Use Edge Computing | |
| automated | Content Delivery Network's (CDN's) |
| automated | Sustainability Commitment |
| manual, qualitative | Local Servers |
| manual, quantitative | Avoid Dynamic Resources |
| 4.11 Use The Lowest Infrastructure Tier Meeting Business Requirements | |
| consulting | Lowest Requirements |
| 4.12 Store Data According To Visitor Needs | |
| consulting | Reduce Redundancy |
| consulting | Expiration Dates |
| consulting | Classify And Tag |
| consulting | Justify Storage |
| consulting | Optimize Logging |
| consulting | Compress Storage |
| 5. Business Strategy And Product Management | |
| 5.1 Have An Ethical And Sustainability Product Strategy | |
| manual, quantitative | Statement Availability |
| manual, qualitative | Achievements And Compliance |
| manual, qualitative | Governance Over Time |
| consulting | Onboarding New Members |
| manual, qualitative | Documentation |
| manual, qualitative | Renewable Showcasing |
| 5.2 Assign A Sustainability Representative | |
| consulting | Ecological Referee |
| 5.3 Raise Awareness And Inform | |
| consulting | Inform And Train |
| consulting | Active Participation |
| 5.4 Communicate The Ecological Impact Of User Choices | |
| manual, qualitative | Impact Communication |
| 5.5 Estimate A Product Or Service's Environmental Impact | |
| consulting | Life-cycle Analysis |
| consulting | Competitor Analysis |
| 5.6 Define Clear Organizational Sustainability Goals And Metrics | |
| manual, qualitative | Sustainability Goals |
| 5.7 Verify Your Efforts Using Established Third-party Business Certifications | |
| consulting | Certification Achievement |
| consulting | Certification Maintenance |
| 5.8 Implement Sustainability Onboarding Guidelines | |
| consulting | Training Materials |
| consulting | Progress Incentivisation |
| consulting | Negative Variables |
| 5.9 Support Mandatory Disclosures And Reporting | |
| consulting | Policies And Practices |
| manual, qualitative | Impact Report |
| manual, qualitative | Standards And Policies |
| consulting | Impact Reduction |
| 5.10 Create One Or More Impact Business Models | |
| consulting | Theory Of Change |
| 5.11 Follow A Product Management And Maintenance Strategy | |
| consulting | Management And Maintenance |
| consulting | Planning Strategy |
| consulting | Resourcing Products |
| consulting | Resource Measurement |
| 5.12 Implement Continuous Improvement Procedures | |
| consulting | Continuous Improvement |
| consulting | Agile Reviews |
| consulting | Iterative Consideration |
| consulting | Functionality Decisions |
| consulting | Security Updates |
| consulting | Skills And Maintenance |
| 5.13 Document Future Updates And Evolutions | |
| consulting | Feature Changes |
| 5.14 Establish If A Digital Product Or Service Is Necessary | |
| consulting | Sustainable Development Goals |
| consulting | Creation Evaluation |
| consulting | Avoid Duplication |
| consulting | Obstacle Consideration |
| 5.15 Determine The Functional Unit | |
| consulting | Life-cycle Assessment |
| 5.16 Create A Supplier Standards Of Practice | |
| consulting | Vetting Potential Partners |
| consulting | Collaborative Measurement |
| consulting | Informative Partner Promotion |
| 5.17 Share Economic Benefits | |
| consulting | Living Wage |
| consulting | Incentivisation |
| consulting | Employee Benefits |
| consulting | Legislation Advocation |
| 5.18 Share Decision-making Power With Appropriate Stakeholders | |
| consulting | Decision-Making |
| 5.19 Use Justice, Equity, Diversity, Inclusion (JEDI) Practices | |
| consulting | JEDI Practices |
| manual, quantitative | Accessibility Policy |
| consulting | JEDI Training |
| consulting | JEDI Improvements |
| consulting | JEDI Legislation |
| 5.20 Promote Responsible Data Practices | |
| manual, qualitative | Privacy Policy |
| consulting | Data Ownership |
| consulting | Data Protection |
| 5.21 Implement Appropriate Data Management Procedures | |
| consulting | Outdated Content |
| manual, qualitative | Data Controllers |
| 5.22 Promote Responsible Emerging Technology Practices | |
| manual, quantitative | Emerging Technologies |
| consulting | Disruptive Technology |
| consulting | Technology Legislation |
| 5.23 Include Responsible Financial Policies | |
| consulting | Fuel Divestment |
| consulting | Responsible Finance |
| 5.24 Include Organizational Philanthropy Policies | |
| consulting | Philanthropy Policy |
| consulting | Success Criterion Voluntary Work |
| 5.25 Plan For A Digital Product Or Service's Care And End-Of-Life | |
| consulting | End-of-life Care |
| 5.26 Include E-waste, Right-to-repair, And Recycling Policies | |
| consulting | E-Waste Policy |
| consulting | Recycling And Repairing |
| consulting | Refurbishment Strategy |
| 5.27 Define Performance And Environmental Budgets | |
| consulting | Environmental Budget |
| consulting | Performance Budget |
| consulting | Measurable Improvements |
| consulting | Capacity And Maintenance |
| 5.28 Use Open Source Tools | |
| consulting | Open Source Policy |
| consulting | Collaboration |
| consulting | Contribution |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment