This reviews how different guidelines / success criteria can be audited, as part of sustyweb#29: Manual VS Automated Testing.
There are three tables to compare:
This reviews how different guidelines / success criteria can be audited, as part of sustyweb#29: Manual VS Automated Testing.
There are three tables to compare:
Guideline-level table, started by @marvil07-adapt, expanded with audit type classification from @thibaudcolas.
Conversion from @thibaudcolas ratings to @marvil07-adapt:
Note for @thibaudcolas automation was only considered possible if the information was publicly available. The potential for internal automation wasn’t considered, except for static analysis (where internal information would help but isn’t a requirement)
Conversion from SC-level rating to guideline-level: use "partial" if the SCs would differ in the yes/no scoring.
How we match up across 93 guidelines:
@marvil07-adapt | @thibaudcolas | Guideline |
---|---|---|
no – internal | no – internal | 2.1 Undertake Systemic Impacts Mapping |
no – internal | no – internal | 2.2 Assess And Research Visitor Needs |
no – internal | no – internal | 2.3 Research Non-visitor's Needs |
no – internal | no – internal | 2.4 Consider Sustainability In Early Ideation |
no – internal | no – internal | 2.5 Account For Stakeholder Issues |
partial | partial | 2.6 Create a Frictionless Lightweight Experience By Default |
partial – editorial | 🚧 Diff: no | 2.7 Avoid Unnecessary Or An Overabundance Of Assets |
partial | 🚧 Diff: no | 2.8 Ensure Navigation And Way-finding Is Well-structured |
partial | 🚧 Diff: no – internal | 2.9 Respect The Visitor's Attention |
no | no | 2.10 Use Recognized Design Patterns |
partial | 🚧 Diff: no | 2.11 Avoid Manipulative Patterns |
no – internal | no – internal | 2.12 Document And Share Project Outputs |
no – internal | no – internal | 2.13 Use A Design System To Prioritize Interface Consistency |
partial | partial | 2.14 Write With Purpose, In An Accessible, Easy To Understand Format |
partial – editorial | partial | 2.15 Take a More Sustainable Approach To Image Assets |
partial – editorial | partial | 2.16 Take a More Sustainable Approach To Media Assets |
no – editorial | no | 2.17 Take a More Sustainable Approach To Animation |
yes | yes | 2.18 Take a More Sustainable Approach To Typefaces |
partial | partial | 2.19 Provide Suitable Alternatives To Web Assets |
no | no | 2.20 Provide Accessible, Usable, Minimal Web Forms |
partial | 🚧 Diff: no | 2.21 Support Non-Graphic Ways To Interact With Content |
no | no | 2.22 Give Useful Notifications To Improve The Visitor's Journey |
partial | partial | 2.23 Reduce The Impact Of Downloadable Or Physical Documents |
no – internal | no – internal | 2.24 Create A Stakeholder-focused Testing & Prototyping Policy |
no – internal | no – internal | 2.25 Conduct Regular Audits, Regression, And Non-regression Tests |
no – internal | no – internal | 2.26 Analyze The Performance Of The Visitor Journey |
no – internal | no – internal | 2.27 Incorporate Value Testing Into Each Major Release-cycle |
no – internal | no – internal | 2.28 Incorporate Usability Testing Into Each Minor Release-cycle |
partial | partial | 2.29 Incorporate Compatibility Testing Into Each Release-cycle |
no – editorial | no - internal | 3.1 Identify Relevant Technical Indicators |
yes | yes | 3.2 Minify Your HTML, CSS, And JavaScript |
partial – internal | 🚧 Diff: yes | 3.3 Use Code-splitting Within Projects |
no – internal | 🔴 Diff: yes | 3.4 Apply Tree Shaking To Code |
partial | partial | 3.5 Ensure Your Solutions Are Accessible |
no – internal | no – internal | 3.6 Avoid Code Duplication |
no | no – internal | 3.7 Rigorously Assess Third-party Services |
partial | 🚧 Diff: yes | 3.8 Use HTML Elements Correctly |
yes – editorial | 🚧 Diff: partial | 3.9 Resolve Render Blocking Content |
partial – semantics | partial | 3.10 Provide Code-based Way-finding Mechanisms |
partial | partial | 3.11 Validate Form Errors And External Input |
partial – editorial, semantics | 🚧 Diff: yes | 3.12 Use Metadata Correctly |
partial – editorial | 🚧 Diff: yes | 3.13 Adapt to User Preferences |
partial | 🚧 Diff: no | 3.14 Develop A Mobile-first Layout |
no | no – internal | 3.15 Use Beneficial JavaScript And Its API's |
partial – internal | 🚧 Diff: yes | 3.16 Ensure Your Scripts Are Secure |
no – internal | 🔴 Diff: yes | 3.17 Manage Dependencies Appropriately |
yes | yes | 3.18 Include Files That Are Automatically Expected |
yes | yes | 3.19 Use Plaintext Formats When Appropriate |
no – internal | 🔴 Diff: yes | 3.20 Avoid Using Deprecated Or Proprietary Code |
no | no – internal | 3.21 Align Technical Requirements With Sustainability Goals |
yes – internal | yes – internal | 3.22 Use The Latest Stable Language Version |
no – internal | no | 3.23 Take Advantage Of Native Features |
no – internal | no | 3.24 Run Fewer, Simpler Queries As Possible |
partial | partial | 4.1 Choose A Sustainable Hosting Provider |
partial – editorial | 🚧 Diff: yes | 4.2 Optimize Browser Caching |
yes | yes | 4.3 Compress Your Files |
no – editorial | 🚧 Diff: partial | 4.4 Use Error Pages And Redirects Carefully |
no – internal | no – internal | 4.5 Limit Usage Of Additional Environments |
no – internal | no – internal | 4.6 Automate To Fit The Needs |
no – editorial | 🚧 Diff: partial | 4.7 Maintain a Relevant Refresh Frequency |
no – internal, editorial | no – internal | 4.8 Be Mindful Of Duplicate Data. |
no – editorial | 🚧 Diff: partial | 4.9 Enable Asynchronous Processing And Communication |
yes | 🚧 Diff: partial | 4.10 Use Edge Computing |
no – internal | no – internal | 4.11 Use The Lowest Infrastructure Tier Meeting Business Requirements |
no – internal, editorial | no – internal | 4.12 Store Data According To Visitor Needs |
no – internal | no | 5.1 Have An Ethical And Sustainability Product Strategy |
no – internal | no – internal | 5.2 Assign A Sustainability Representative |
no – internal, internal | no – internal | 5.3 Raise Awareness And Inform |
no – internal | no | 5.4 Communicate The Ecological Impact Of User Choices |
partial – internal | 🚧 Diff: no – internal | 5.5 Estimate A Product Or Service's Environmental Impact |
no – internal | no | 5.6 Define Clear Organizational Sustainability Goals And Metrics |
no – internal | no – internal | 5.7 Verify Your Efforts Using Established Third-party Business Certifications |
no – internal | no – internal | 5.8 Implement Sustainability Onboarding Guidelines |
no – internal | no | 5.9 Support Mandatory Disclosures And Reporting |
no – internal | no – internal | 5.10 Create One Or More Impact Business Models |
no – internal | no – internal | 5.11 Follow A Product Management And Maintenance Strategy |
no – internal | no – internal | 5.12 Implement Continuous Improvement Procedures |
no – internal | no – internal | 5.13 Document Future Updates And Evolutions |
no – internal | no – internal | 5.14 Establish If A Digital Product Or Service Is Necessary |
no – internal | no – internal | 5.15 Determine The Functional Unit |
no – internal | no – internal | 5.16 Create A Supplier Standards Of Practice |
no – internal | no – internal | 5.17 Share Economic Benefits |
no – internal | no – internal | 5.18 Share Decision-making Power With Appropriate Stakeholders |
no – internal | 🚧 Diff: partial | 5.19 Use Justice, Equity, Diversity, Inclusion (JEDI) Practices |
no – internal | no | 5.20 Promote Responsible Data Practices |
no – internal | no – internal | 5.21 Implement Appropriate Data Management Procedures |
no – internal | 🚧 Diff: partial | 5.22 Promote Responsible Emerging Technology Practices |
no – internal | no – internal | 5.23 Include Responsible Financial Policies |
no – internal | no – internal | 5.24 Include Organizational Philanthropy Policies |
no – internal | no – internal | 5.25 Plan For A Digital Product Or Service's Care And End-Of-Life |
no – internal | no – internal | 5.26 Include E-waste, Right-to-repair, And Recycling Policies |
partial – internal | 🚧 Diff: no – internal | 5.27 Define Performance And Environmental Budgets |
no – internal | no – internal | 5.28 Use Open Source Tools |
Expected audit types for all success criteria, from the perspective of a consultant WSG specialist, with an audit methodology similar to WCAG-EM.
Types of audits classified from most to least automate-able across 232 SCs:
jsx-a11y/alt-text
Audit type | Success Criterion |
---|---|
2. User-Experience Design | |
2.1 Undertake Systemic Impacts Mapping | |
consulting | External Variables |
2.2 Assess And Research Visitor Needs | |
consulting | Identify And Define |
consulting | Visitor Constraints |
consulting | Barriers And Access |
consulting | Barrier Removal |
2.3 Research Non-visitors Needs | |
consulting | Non-Human Impact |
2.4 Consider Sustainability In Early Ideation | |
consulting | Wireframes And Prototypes |
2.5 Account For Stakeholder Issues | |
consulting | Human-Centered Brainstorming |
consulting | Ecological Brainstorming |
2.6 Create a Frictionless Lightweight Experience By Default | |
consulting | Performance By Default |
manual, quantitative | Efficient Paths |
manual, qualitative | Patterns For Efficiency |
manual, qualitative | Distraction-Free Design |
manual, qualitative | Eliminate The Non-Essential |
manual, quantitative | User-Initiated Actionable Content |
2.7 Avoid Unnecessary Or An Overabundance Of Assets | |
manual, qualitative | Decorative Design |
2.8 Ensure Navigation And Way-finding Is Well-structured | |
manual, qualitative | Navigation And Search |
manual, qualitative | Navigable Sitemaps |
manual, qualitative | New Content |
2.9 Respect The visitors Attention | |
manual, qualitative | Respecting Attention |
consulting | Avoid Distraction |
manual, qualitative | Avoid Attention-keeping |
2.10 Use Recognized Design Patterns | |
manual, qualitative | Design Patterns |
2.11 Avoid Manipulative Patterns | |
manual, qualitative | Dark Patterns |
manual, qualitative | Using Advertisements |
manual, qualitative | Page Tracking |
2.12 Document And Share Project Outputs | |
consulting | Deliverables Reusability |
consulting | Deliverables Documentation |
2.13 Use A Design System To Prioritize Interface Consistency | |
consulting | Design System |
2.14 Write With Purpose, In An Accessible, Easy To Understand Format | |
manual, qualitative | Write Clearly |
manual, qualitative | Content Formatting |
static analysis | Search Engine Optimization (SEO) |
2.15 Take a More Sustainable Approach To Image Assets | |
automated | Need For Images |
automated | Optimize Images |
static analysis | Lazy Loading |
manual, qualitative | Sizing And Deactivation |
consulting | Management And Usage |
2.16 Take a More Sustainable Approach To Media Assets | |
automated | Need For Media |
automated | Optimize Media |
manual, qualitative | Labels And Choice |
consulting | Management And Usage |
2.17 Take a More Sustainable Approach To Animation | |
manual, qualitative | Need For Animation |
manual, qualitative | Avoid Overburdening |
manual, qualitative | Control Animation |
2.18 Take a More Sustainable Approach To Typefaces | |
static analysis | Default Typefaces |
static analysis | Font Optimization |
2.19 Provide Suitable Alternatives To Web Assets | |
manual, quantitative | Open Formats |
manual, quantitative | Font Subsetting |
automated | Alternative Text |
manual, quantitative | Audio Alternatives |
manual, quantitative | Video Alternatives |
2.20 Provide Accessible, Usable, Minimal Web Forms | |
manual, qualitative | Form Simplicity |
manual, qualitative | Form Functionality |
2.21 Support Non-Graphic Ways To Interact With Content | |
manual, qualitative | Alternative Interactions |
2.22 Give Useful Notifications To Improve The visitors Journey | |
manual, qualitative | Notification Justification |
manual, qualitative | Notification Control |
manual, qualitative | Prompts And Responses |
2.23 Reduce The Impact Of Downloadable Or Physical Documents | |
manual, quantitative | Printing Documents |
manual, quantitative | Optimize Documents |
manual, quantitative | Labels And Choice |
2.24 Create A Stakeholder-focused Testing & Prototyping Policy | |
consulting | New Features And Perspectives |
consulting | Resourcing And Viability |
consulting | Training And Onboarding |
consulting | Testing And Validation |
2.25 Conduct Regular Audits, Regression, And Non-regression Tests | |
consulting | Regular Issue Testing |
consulting | Non-Regression Tests |
consulting | Regression Tests |
2.26 Analyze The Performance Of The Visitor Journey | |
consulting | Measurement And Compliance |
2.27 Incorporate Value Testing Into Each Major Release-cycle | |
consulting | Usage Changes |
2.28 Incorporate Usability Testing Into Each Minor Release-cycle | |
consulting | Usability Testing |
2.29 Incorporate Compatibility Testing Into Each Release-cycle | |
consulting | Compatibility Policy |
consulting | Maintaining Compatibility |
manual, quantitative | Frequent Testing |
consulting | Mobile Friendly |
consulting | Progressive Web Application's (PWA)'s |
3. Web Development | |
3.1 Identify Relevant Technical Indicators | |
consulting | Performance Goals |
3.2 Minify Your HTML, CSS, And JavaScript | |
automated | Minify Code |
3.3 Use Code-splitting Within Projects | |
automated | Code Splitting |
3.4 Apply Tree Shaking To Code | |
automated | Remove Redundancy |
3.5 Ensure Your Solutions Are Accessible | |
automated | Accessibility Compliance |
static analysis | Enhancing For Accessibility |
consulting | Electronic Inequalities |
manual, qualitative | Carbon Aware Design |
3.6 Avoid Code Duplication | |
consulting | Remove Or Simplify |
consulting | Iteration Over Recreation |
consulting | Organize Code Arrangement |
3.7 Rigorously Assess Third-party Services | |
consulting | Assess Third-parties |
consulting | Third-party Implementation |
consulting | Libraries And Frameworks |
consulting | Self-Hosting |
consulting | Avoiding Dependency |
3.8 Use HTML Elements Correctly | |
automated | Semantic Code |
static analysis | Avoid Non-standard Code |
manual, qualitative | Custom Code |
3.9 Resolve Render Blocking Content | |
automated | Asynchronous Code |
manual, qualitative | Priority Loading |
3.10 Provide Code-based Way-finding Mechanisms | |
automated | Metadata And Microdata |
automated | Search Engines |
manual, qualitative | Accessibility Aids |
3.11 Validate Form Errors And External Input | |
manual, qualitative | Error Validation |
manual, qualitative | Label Elements |
automated | Allow Paste |
3.12 Use Metadata Correctly | |
automated | Required Elements |
automated | Meta Tags |
automated | Structured Data |
3.13 Use CSS Preference And Media Queries | |
automated | CSS Preference Queries |
3.14 Develop A Mobile-first Layout | |
manual, qualitative | Mobile-First |
manual, qualitative | Responsive Design |
manual, qualitative | Alternative Browsing |
3.15 Use Beneficial JavaScript And Its API's | |
consulting | Beneficial JavaScript |
manual, qualitative | API Requests |
3.16 Ensure Your Scripts Are Secure | |
automated | Script Security |
3.17 Manage Dependencies Appropriately | |
automated | Dependency Management |
automated | Dependency Necessity |
automated | Dependency Updates |
3.18 Include Files That Are Automatically Expected | |
automated | Expected File Formats |
3.19 Use Plaintext Formats When Appropriate | |
automated | Beneficial File Formats |
3.20 Avoid Using Deprecated Or Proprietary Code | |
automated | Deprecated Code |
automated | Outdated Code |
3.21 Align Technical Requirements With Sustainability Goals | |
consulting | Identify Requirements |
consulting | Optimized Methodology |
consulting | Expandability Considerations |
consulting | Interface Impact |
3.22 Use The Latest Stable Language Version | |
automated | Versioning |
3.23 Take Advantage Of Native Features | |
manual, qualitative | Native Over Custom |
3.24 Run Fewer, Simpler Queries As Possible | |
manual, qualitative | Database Queries |
4. Hosting, Infrastructure And Systems | |
4.1 Choose A Sustainable Hosting Provider | |
consulting | Monitor Metrics |
consulting | Equipment Longevity |
consulting | Recycling Waste |
automated | Renewable Electricity |
consulting | Remaining Emissions |
4.2 Optimize Browser Caching | |
automated | Server-side Caching |
automated | Offline Access |
4.3 Compress Your Files | |
automated | Server-side Compression |
automated | Media Compression |
4.4 Use Error Pages And Redirects Carefully | |
manual, quantitative | Error Pages |
manual, qualitative | Redirection |
4.5 Limit Usage Of Additional Environments | |
consulting | Unused Environments |
4.6 Automate To Fit The Needs | |
consulting | Automate Tasks |
consulting | Necessitate Tasks |
consulting | Automated Scaling |
consulting | Security Tooling |
4.7 Frequency For Refresh Is Relevant To Visitor Needs | |
manual, quantitative | Refresh Frequency |
4.8 Be Mindful Of Duplicate Data. | |
consulting | Data Backups |
4.9 Enable Asynchronous Processing And Communication | |
consulting | Batch Processing |
automated | Protocol Usage |
4.10 Use Edge Computing | |
automated | Content Delivery Network's (CDN's) |
automated | Sustainability Commitment |
manual, qualitative | Local Servers |
manual, quantitative | Avoid Dynamic Resources |
4.11 Use The Lowest Infrastructure Tier Meeting Business Requirements | |
consulting | Lowest Requirements |
4.12 Store Data According To Visitor Needs | |
consulting | Reduce Redundancy |
consulting | Expiration Dates |
consulting | Classify And Tag |
consulting | Justify Storage |
consulting | Optimize Logging |
consulting | Compress Storage |
5. Business Strategy And Product Management | |
5.1 Have An Ethical And Sustainability Product Strategy | |
manual, quantitative | Statement Availability |
manual, qualitative | Achievements And Compliance |
manual, qualitative | Governance Over Time |
consulting | Onboarding New Members |
manual, qualitative | Documentation |
manual, qualitative | Renewable Showcasing |
5.2 Assign A Sustainability Representative | |
consulting | Ecological Referee |
5.3 Raise Awareness And Inform | |
consulting | Inform And Train |
consulting | Active Participation |
5.4 Communicate The Ecological Impact Of User Choices | |
manual, qualitative | Impact Communication |
5.5 Estimate A Product Or Service's Environmental Impact | |
consulting | Life-cycle Analysis |
consulting | Competitor Analysis |
5.6 Define Clear Organizational Sustainability Goals And Metrics | |
manual, qualitative | Sustainability Goals |
5.7 Verify Your Efforts Using Established Third-party Business Certifications | |
consulting | Certification Achievement |
consulting | Certification Maintenance |
5.8 Implement Sustainability Onboarding Guidelines | |
consulting | Training Materials |
consulting | Progress Incentivisation |
consulting | Negative Variables |
5.9 Support Mandatory Disclosures And Reporting | |
consulting | Policies And Practices |
manual, qualitative | Impact Report |
manual, qualitative | Standards And Policies |
consulting | Impact Reduction |
5.10 Create One Or More Impact Business Models | |
consulting | Theory Of Change |
5.11 Follow A Product Management And Maintenance Strategy | |
consulting | Management And Maintenance |
consulting | Planning Strategy |
consulting | Resourcing Products |
consulting | Resource Measurement |
5.12 Implement Continuous Improvement Procedures | |
consulting | Continuous Improvement |
consulting | Agile Reviews |
consulting | Iterative Consideration |
consulting | Functionality Decisions |
consulting | Security Updates |
consulting | Skills And Maintenance |
5.13 Document Future Updates And Evolutions | |
consulting | Feature Changes |
5.14 Establish If A Digital Product Or Service Is Necessary | |
consulting | Sustainable Development Goals |
consulting | Creation Evaluation |
consulting | Avoid Duplication |
consulting | Obstacle Consideration |
5.15 Determine The Functional Unit | |
consulting | Life-cycle Assessment |
5.16 Create A Supplier Standards Of Practice | |
consulting | Vetting Potential Partners |
consulting | Collaborative Measurement |
consulting | Informative Partner Promotion |
5.17 Share Economic Benefits | |
consulting | Living Wage |
consulting | Incentivisation |
consulting | Employee Benefits |
consulting | Legislation Advocation |
5.18 Share Decision-making Power With Appropriate Stakeholders | |
consulting | Decision-Making |
5.19 Use Justice, Equity, Diversity, Inclusion (JEDI) Practices | |
consulting | JEDI Practices |
manual, quantitative | Accessibility Policy |
consulting | JEDI Training |
consulting | JEDI Improvements |
consulting | JEDI Legislation |
5.20 Promote Responsible Data Practices | |
manual, qualitative | Privacy Policy |
consulting | Data Ownership |
consulting | Data Protection |
5.21 Implement Appropriate Data Management Procedures | |
consulting | Outdated Content |
manual, qualitative | Data Controllers |
5.22 Promote Responsible Emerging Technology Practices | |
manual, quantitative | Emerging Technologies |
consulting | Disruptive Technology |
consulting | Technology Legislation |
5.23 Include Responsible Financial Policies | |
consulting | Fuel Divestment |
consulting | Responsible Finance |
5.24 Include Organizational Philanthropy Policies | |
consulting | Philanthropy Policy |
consulting | Success Criterion Voluntary Work |
5.25 Plan For A Digital Product Or Service's Care And End-Of-Life | |
consulting | End-of-life Care |
5.26 Include E-waste, Right-to-repair, And Recycling Policies | |
consulting | E-Waste Policy |
consulting | Recycling And Repairing |
consulting | Refurbishment Strategy |
5.27 Define Performance And Environmental Budgets | |
consulting | Environmental Budget |
consulting | Performance Budget |
consulting | Measurable Improvements |
consulting | Capacity And Maintenance |
5.28 Use Open Source Tools | |
consulting | Open Source Policy |
consulting | Collaboration |
consulting | Contribution |