community metrics thread from 4 jan 18 dd&i meeting

gistfile1.txt

:three: Community metrics - should this group be working towards specific, measurable goals? What metrics would be meaningful and how would we collect them? The issue has more ideas and prompts for discussion: `https://www.drupal.org/project/diversity/issues/2929249`
What are the things we (as DD&I) want to have an impact on and  how can we measure those? What is most meaningful to us? @agentrickard is with us to discuss further.


83 replies
agentrickard [44 minutes ago] 
The first question is one of scope: Is this issue about metrics for Drupal, or about metrics for DD&I initiatives?

rubyji [43 minutes ago] 
I’m not against the idea of measuring, but this can be SO incredibly fraught with complexity and unintended consequences, so I’d approach this cautiously. Especially the idea of having numeric “goals.”


agentrickard [42 minutes ago] 
@rubyji please read the post about the dirreence between qualitative and quantitiatve metrics

agentrickard [42 minutes ago] 
https://www.drupal.org/project/diversity/issues/2929249#comment-12400490
Drupal.org
Meaningful metrics?
Original discussion -- https://github.com/drupaldiversity/administration/issues/73
Dec 7th at 10:21 AM

agentrickard [41 minutes ago] 
difference :slightly_smiling_face:

rubyji [41 minutes ago] 
Yes, I read your post yesterday.

agentrickard [41 minutes ago] 
I would agree

agentrickard [41 minutes ago] 
thanks

agentrickard [40 minutes ago] 
i think organizations should have goals, and individuals should have what, opinions / feelings / attitudes. And I think separating those two sets of metrics would give us clarity.

agentrickard [40 minutes ago] 
I don't really have a strong opinion or expertise here. I'm really trying to facilitate so others don't have to

catch [40 minutes ago] 
So I think metrics can show how bad things are, but they don’t necessarily reflect things being good. And that’s where goals get extremely tricky.

agentrickard [38 minutes ago] 
agreed.

agentrickard [38 minutes ago] 
Can we first define the scope of this conversation: Are we talking about Drupal metrics or DD&I metrics?

alannaburke [37 minutes ago] 
would it maybe be wise to start with measuring, see where we are, and then think about goals?

drnikki [37 minutes ago] 
I think it’s simplest to define DD&I metrics (and using metrics in the _broadest_ sense here)

catch [37 minutes ago] 
It overlaps a bit. I posted a long comment on Dries’ blog post on contribution statistics, because I felt the statistical usage was bad.

rachel_norfolk [37 minutes ago] 
I’m interested as there might be some +ve movement on the “notorious gender field” in the near-ish future but for which the Association will need the guidance and assistance of groups such as this to get right. When we’re ready, we’ll come calling!

rachel_norfolk [37 minutes ago] 
(And we’re not _quite_ ready yet)

agentrickard [36 minutes ago] 
Let me recap what I just heard in the form of a motion

agentrickard [36 minutes ago] 
"The metrics initiative should focus on DD&I efforts and be prepared so share those outcomes and tools with the broader community"

agentrickard [35 minutes ago] 
Is that accurate?

catch [35 minutes ago] 
hmm and he hasn’t published my comment yet, can expand if it’s useful but possibly not right now.

drnikki [35 minutes ago] 
I think that’s close.

catch [34 minutes ago] 
The only thing I’d say is that something like limitations in contribution credit could impact what DD&I stats it’s possible to collect and then impact how that’s presented (which is very much the case for https://dri.es/who-sponsors-drupal-development-2017 in multiple ways).
dri.es
Who sponsors Drupal development? (2016-2017 edition)
An in-depth analysis of how Drupal's development was sponsored between July 1, 2016 and June 30, 2017. (138 kB)

agentrickard [33 minutes ago] 
@catch I think that issue is preamture to the question at hand, though I agree that "contribution" is problematic

catch [33 minutes ago] 
But also contribution credits themselves can be fraught so it doesn’t mean we should get into it.


catch [33 minutes ago] 
Yep.

drnikki [33 minutes ago] 
“The “is DD&I effective” initiative may include metrics, and, regardless of the outcome, should include information/takeaways that we can share with the broader community”


drnikki [32 minutes ago] 
i worry that a premature focus on even the word ‘metrics’ is going to encourage us to miss things


drnikki [32 minutes ago] 
But i’m 1,000% stoked that you’re here and thinking about this in such an organized way @agentrickard.  It’s really helpful to have someone dig in and work through this

agentrickard [31 minutes ago] 
ok, so motion #1 is that we focus on measuring the effectiveness / effects of DD&I

agentrickard [31 minutes ago] 
Yes (+)


agentrickard [31 minutes ago] 
No (+)

rubyji [29 minutes ago] 
I don’t understand what is happening here. We have never have never had a motion in a meeting before.

agentrickard [29 minutes ago] 
How else do we record that decisions were made?


rubyji [29 minutes ago] 
We don’t usually give an official blessings to efforts that our members do, even though we love them.


rubyji [28 minutes ago] 
The question is how do we even make decisions, not how we record them.


agentrickard [28 minutes ago] 
:slightly_smiling_face:

agentrickard [28 minutes ago] 
I admit to a very phallogocentric method of decision making

agentrickard [28 minutes ago] 
:slightly_smiling_face:

drnikki [28 minutes ago] 
The official/unofficial blessing is that we keep talking about it and putting it on the agenda.  So, right now, @agentrickard is our defacto “success measurer” person.  and we’re going to keep asking you to talk about the ways you’re thinking about measuring success.

rubyji [28 minutes ago] 
I don’t think we should be making firm decisions that can have a big impact based on just a random assortment of who is here today.

agentrickard [27 minutes ago] 
ok

agentrickard [27 minutes ago] 
I don't quite know how to work within that constraint

drnikki [27 minutes ago] 
To be fair, we hardly ever make firm decisions, haha. :slightly_smiling_face: it takes a super long time because we do our best to have a consensus democracy in this space.



drnikki [27 minutes ago] 
Well, here’s a question then - @agentrickard what do you need / what would you like?

agentrickard [26 minutes ago] 
Right now, I'm just trying to get agreement on scope -- DD&I or Drupal overall.

rubyji [26 minutes ago] 
Most of the initiatives here are organized by one or two people leading and pushing them forward, with regular check ins, and documentation in d.o (formerly github).


agentrickard [26 minutes ago] 
because the original thread was about Drupal as a whole

agentrickard [26 minutes ago] 
that doesn't sound very democratic

drnikki [25 minutes ago] 
Okay, our decision-making process aside. :slightly_smiling_face:



agentrickard [25 minutes ago] 
I am happy to push this one forward if we can agree on the scope


rachel_norfolk [25 minutes ago] 
So, you’re asking whether you should be measuring whether efforts are having an effect upon the whole Drupal COmmunity. That’s right, @agentrickard ?


drnikki [25 minutes ago] 
1. does Drupal as a community need a metrics plan for diversity and inclusion? I’d say yes to that.


drnikki [24 minutes ago] 
2. Are we the best group to do that?  I’m not sure.


agentrickard [24 minutes ago] 
@rachel_norfolk that is one option. the other option is setting measurements for the entire community


drnikki [24 minutes ago] 
3. Does DD&I need a way to figure out if it’s existence _matters_?  yes.


agentrickard [24 minutes ago] 
I can agree to all of that and agree that #3 is the part within our control

drnikki [24 minutes ago] 
3.1. Does Dd&I need a way to figure out how to make its existence _matter more_? yes.


agentrickard [21 minutes ago] 
In the d.o. thread, I propose that one approach is to gather qualitative metrics about people's experience with the community (primarily at events) and try to track that over time to see if we are improving. I still think that's valid, though it has risks

rachel_norfolk [21 minutes ago] 
Can I suggest 3.2 _matters to whom?_ because that varies by person


rubyji [17 minutes ago] 
I don’t see the point in measuring something if it’s totally subjective. The point should be that it matters _to the community_. Like that it makes Drupal better and makes the community more successful.

alannaburke [17 minutes ago] 
i don’t want to give anyone the opportunity to say that DD&I doesn’t matter or doesn’t make a difference.

agentrickard [16 minutes ago] 
I don't see how you can do objective measurements without setting goals

drnikki [15 minutes ago] 
well, so part of this is framing - if you’re looking at something from an intersectional perspective (which you can also think of as matrix thinking) there isn’t a ton of “objective” measurements.

drnikki [15 minutes ago] 
It feels to me like we’re a bit in the weeds here.

drnikki [15 minutes ago] 
If we can agree on a _question to ask_ then we can answer it from a quantitative and/or qualitative  and/or narrative position

drnikki [13 minutes ago] 
So (and this is just an example, not saying this is THE question), but if we asked “Since the formation of DD&I is the drupal community safer for female-identified individuals?” we could answer that by looking at the number of F-i-i who commit code, or who speak (quantitative) or we could talk to F-i-i and ask how they’ve handled CoC violations. or we could collect stories of  feeling safe or not safe


agentrickard [13 minutes ago] 
my thought here was to focus on he _inclusion_ part of the initiative as well, which this example gets at

drnikki [11 minutes ago] 
and i love that. the _inclusion_ bit is the toughest to “measure” and so I think if we can make some structured progress in describing inclusion shifts over time, we’ll be doing a great service to other communities (and the broader community as it works to figure this stuff out)


agentrickard [10 minutes ago] 
so how do we summarize this conversation?


drnikki [8 minutes ago] 
That’s a great question.  :slightly_smiling_face:


drnikki [7 minutes ago] 
I think the next step is to decide on a few, _very specific_ questions to ask under our new limited scope of: “What kind of difference, if any, does DD&I make in the Drupal community?”


drnikki [6 minutes ago] 
With the goal of doing more of what helps and less of what doesn’t so that we can focus our energy.


drnikki [6 minutes ago] 
For me the big takeaways were to not stress about how to measure or even what to measure, but to focus on getting agreement on what to ask and why.


drnikki [6 minutes ago] 
That will guide next steps.


agentrickard [6 minutes ago] 
and are we asking individuals or teams (e.g. conference organizers)? I assume individuals


drnikki [4 minutes ago] 
mmm, individuals, yes.


agentrickard [3 minutes ago] 
OK. Let's sit with that a bit and maybe bring it back next meeting?


agentrickard [3 minutes ago] 
I have to go eat


drnikki [3 minutes ago] 
yes, totally.  I’ll add it to the next agenda. :slightly_smiling_face:


agentrickard [2 minutes ago] 
OK. Thank you to everyone on the thread. This was very helpful for me.