Measuring and metrics

Outcomes-orientation

What's this about?

Understanding outcomes involves measuring progress, which can be notoriously difficult with software development. In this lesson, we explore what it means to assess progress, and whether a team can demonstrate progress against a set of reasonable metrics and baselines.

Lesson outline

• Ordering: What do you think? (15m, solo)
• Reading: What the experts say (30m, solo)
• Viewing: Counterpoint on measurement (45m, small group)
• Reflection (10m, solo)
• Sharing experience (30m, small group)
• Discuss in community (1h, group)

At some level, measuring progress on a large, long-running software project is an impossible task.

Given that, it means that is is a huge space for vendors and states to claim that progress is being made even if progress is not being made.

This lesson looks at some frameworks for assessing the health of a program and encourages you to develop a “questioning state of mind” when it comes to evaluating how states and vendors measure progress and success.

Ordering: What do you think? (15m, solo)

Jama Software has nine metrics that they think relate to software team speed and productivity. We offer these as food for thought, and are not suggesting that they are “the best” or “the only” ways to think about software team productivity.

Give the article a skim. Make sure to note questions you have about these metrics as you read; we have to admit, it’s not a very… verbose… article. Then come back and re-order the metrics (by dragging-and-dropping) so that the metric that you think would tell you the most about a team’s product quality is at the top and the metric that you think would tell you the least is at the bottom.

Note your top two and bottom two in your notebook for later discussion, as well as why you thought these were the top two and bottom two.

Reading: What the experts say (30m, solo)

Stackify’s blog has an interesting set of interviews with experts in software development and the management of software projects.

Below are two random experts from the article. (Click to generate two new names!) Jot these down.

Read what those two experts have to say in the Stackify article. After you’ve read and noted the highlights of what they have to say, compare and contrast. If you were approaching the evaluation of your state’s projects, which of these experts do you think are offering you better advice? Why? Which do you think would not be useful to you? Again, why?

Record your thoughts in your notebook. We’re going to come back together to report out on what you gathered from these experts and also share out what you thought would actually be useful to you in your context.

Viewing: Counterpoint on measurement (45m, small group)

You began by taking a very terse list of success metrics, and ordering them by importance. Then you read the statements of two experts and compared and contrasted what they had to say about software quality and measuring the progress of software teams. Now we’re going to critique one developer’s (PM’s?) view on measuring software quality.

Or: there’s nothing like a little bit of devil’s advocacy to spice things up.

First, schedule 30m that you can get together with a colleague. In preparation for that meeting, watch this video once. This way, you’re familiar with the content before diving into discussion.

Now, when you get together with your colleague, have a viewing party. This is where you watch a video together and comment in realtime. Normally, people install some software on their computers so that one person can control the start-and-stop of the video on both machines… but, we can’t do that (because gov’t computers). You could click the gear (the settings icon in the YouTube player) and reduce the quality of the video (to reduce bandwidth requirements), and have one person screenshare while another controls playback (it might work). Or you can just play it on your separate machines and shout “stop!” and “start!” at each-other. Whatever works.

This video was chosen because:

1. It is short (8m long).
2. It is debateable.
3. Claims made are likely representative of vendor claims.

This is an exercise in calling bullshit. Imagine this is a vendor describing their software quality measurement process. You and your colleague were unable to make the presentation, so it was recorded for you to watch later. Therefore, you have the opportunity to comment and critique without having to respond in real-time to the vendor.

It is easy to have “pretender syndrome,” where you wonder if you don’t know enough about software projects to comment meaningfully. One way to overcome that is to develop foundational skills and knowledge (we’re doing that in this course). However, talking about doing things is not the same as doing things. This roleplay activity is just that: instead of talking about riding a bike, we’re actually getting up on a bike and practicing.

Have fun, call bullshit, and if you need to, take some time to ask “why?” There’s learning in exploring why a colleague thinks something has a smell. And, there’s learning in your own explanations.

Reflection (10m, solo)

When you’re done, spend 10 minutes making notes in your journal. Specifically, make note of two or three things from your “calling bullshit” session where you note what you were particularly concerned about during the presentation, and why.

Sharing experience (30m, small group)

Meet with your small group and connect what you learned in this lesson to situations you’ve seen with your state projects. Consult the notes you took throughout the lesson and try to link them to a story that you can tell about a particular project. It’s probably useful to do some brainstorming on this before you meet with your small group to trade stories.

When you get together with your small group:

1. Share your stories with each other.
2. Figure out which ones are the best candidates for a case study or use case that would be helpful to share with other state officers.
3. As a group, choose useful stories and write notes on how they link up with the concepts shared.
   • Include in your notes:
     • When did this story take place?
     • What were the events or background leading up to this story?
     • How did this story demonstrate an ideal or non-ideal situation?
     • What specific principles from the lesson does this story illustrate?
     • If the story shows an ideal, what were the conditions that made it work? How did it fit with the principles shared in the lesson?
     • If the story shows a non-ideal, what could have changed to make it better?
4. Share these notes with the larger group when you meet.
5. After discussion with the larger group, document these stories and their connections to the lesson to help other state officers understand how this lesson’s concepts apply to their work.

Discuss in community (1h, group)

These questions are all meant to help you get at this one dimension of the rubric:

Come together with your colleagues for a conversation.

You can click on the timers below to help keep yourselves on track.

1. Check in. (5m timer) First, check in with each-other. How is everyone doing? Take a moment to share something positive from the week, either at work or at home.
2. RapidShare™ Prep (1m timer) Take one minute to prepare what you want to say about your small group conversation. Specifically, note two things: a single word, and a single sentence.
3. RapidShare™ Word. (2m timer) Next, take three minutes to slowly go around the group, and share out the one word that captured your reflections and thoughts from your small group activity. What one word captured your learning/insights?
4. RapidShare™ Sentence: (8m timer) Now, go around the group again. Use the template “My one-word summary was __, and my one sentence summary would be __,” inserting your one word and one sentence in the template.
5. Reflecting. (2m timer) Now, pause. For two minutes, make notes on your own. Specifically, brainstorm one, possibly two questions you might want to ask the group about the learning of this lesson in light of what everyone shared. Perhaps you want to probe deeper into one person’s comments, or perhaps you have a larger/”big picture” question for the group against the overall backdrop.
6. Discuss. (30m timer) Let the conversation go where it does. Explore each-other’s questions. Make sure everyone has a chance to join in the conversation, through questions and comments. If you have to, use cards or popsicle sticks with everyone’s name on them to make sure everyone takes part.
7. Wrap up. (5m timer)

In the guides

This lesson is the beginning of a journey. If you're interested in learning more, there's material in the 18F Derisking Guide that you'll want to check out.

From the Federal Field Guide:

• Tightly scoped software development
• Invest in tech incrementally
• Evaluate proposals based on best practices
• Oversight of outcomes

From the State Software Budgeting Handbook:

• Demos, not memos
• Measure success iteratively
• Fund systems, not monoliths
• Ask technical questions

Wrapup (5m, solo)

Take a few minutes to share your reflections on this lesson.