Attention!

The content on this site is a materials pilot. It represents neither changes to existing policy nor pending new policies. THIS IS NOT OFFICIAL GUIDANCE.

Questioning the teams


State capacity

Ask detailed questions about technology, program goals, end users, etc.

Pay attention to who answers each question.

  • Bad: Team members can't answer questions about the project or issues in plain language.
  • Meh: Team members rely on vendors to answer questions, but answers are in plain language.
  • Good: All team members (including IT and procurement) can speak in plain language about the project.

What's this about?

To have a functioning software team, you gotta get folks on the same page. This lesson helps you figure out whether everyone on the team knows the story and how well they know it.

Lesson outline

Reflection: What makes a team work? (15m, pair)

Timer: (15m timer)

First, take some time to think through what makes a good functioning team. Grab a colleague and brainstorm some good signs on teams.

Discuss when and how team members relate to each other. Debate the value of everyone on the team understanding each other’s piece of the pie vs. team members working away in their own silos. In what cases do team members have to know about each other’s work, and how much should they know?

Take notes and be ready to share.

Asking questions of teams

Read: Asking questions of software project teams (45m, solo)

It can be hard to know where to ask sofware teams questions that tease out their team and project health. There’s a few items that may help you knmow what to ask.

  1. Asking Questions of Software Project Teams
  2. Asking Technical Questions of Agencies

For bonus reading on evaluating projects, the whole 18F State Field Guide is worth a look!

As you read these, take notes on anything that jumped out at you. Can you see yourself using this with any of your projects? How do you think it would go?

Paying attention to who answers what (10m, solo)

We’re going to go a bit further here and have some subjects to think through for your questions. As you think these through, remember you’ll need to note a few things. Does the vendor give all the answers? Does the state know what they’re buying and the terms of their vendor contract?

Subjects to ask about:

Technology

  • Who hosts the software tool?
  • Does the state have access to the code?
  • Program and team goals
  • What are the main goals of this tool and how do they solve the problem?

End Users

  • What do we know about the users of the tool?
  • How are we building and improving the tool to fit our desired user outcomes?

Vendor

  • What’s the duration of the relationship with the vendor? Under what circumstances does it end?
  • Which departments have to know what about the contract and how it affects the work (IT, Procurement, software…do they all understand it?)?

Watch: Unpacking the jargon (15m, solo)

I’ll borrow a quote from a previous lesson on state capacity :

To assess project health, you’ll need to be able to keep up with the jargon. And, frankly, jargon is a great way to bullshit someone. If I want you to 1) think I know what I’m talking about, but 2) confuse you along the way, I’m going to use jargon and words that I think you don’t know well. If a product manager or project manager wants to mislead you regarding the health of a project, or otherwise mislead you as to how things are going, they’re going to hide behind the words of their trade, and try and hide the realities of a project’s health in the details.”

Another way to get around jargon is to ask the person speaking to explain it to you like you’re a small child. Or a golden retriever. We’re sharing this movie clip for inspiration, but definitely not endorsing this using this sort of behavior to ask for simplified language.

The point is that you can feel comfortable asking someone to back up and explain things back simply. If they can’t use plain language to talk about the work, question if they’re possibly trying to hide something behind a lot of jargon.

Roleplay: Play the part (1h, full group)

To close out this lesson, come together with your learning cohort.

  1. Checkin. (10m timer) How is everyone? Take a moment to go around the group, and offer up something that brought you joy this past week. Start with birthdays in December, and work backwards through the year.
  2. Prep. (3m timer) Now divide the group, so that part of the group is people who are on the State side — a developer, a procurement officer, and a project manager for the State. The other half the group represents the SO asking questions of the group. The State group should put themselves in the place of their state’s project, and imagine themselves as a program manager (or procurement officer, or developer) for that project. The SO group should take this time and imagine they are prepping for a conversation with these state team members to work on assessing project health. Take three minutes to prep for the roleplay.
  3. Breakout I. (5m timer) Break into groups. Each pair should have one state officer and each of the state team members. Have the state officer talking each of the team members, working to understand the health of the project and asking the team members to explain the project. If it helps, feel free to choose questions to ask from the two guides mentioned earlier: Asking Questions of Software Project Teams and Asking Technical Questions of Agencies
  4. Debrief I. (15m timer) After 5 minutes, come back to your large group. As a group, discuss the roleplays.
    • For the SOs: What questions were particularly useful? What words/behaviors did you note from the state team members that were encouraging or inspired confidence?
    • For the State team members: How did getting into the state’s head shape your thinking about how the state may explain projects to you? What mindset did you bring to the conversation? Was it collaborative? Evasive? Why?
  5. Wrap. (5m timer) Take a moment, before leaving, to share what your favorite moments were from the learning activity. Go around the group, and offer one kudo to your roleplay partner—something they did that you thought was particularly wonderful, fun, or insightful.

Regarding Roleplays

Roleplays are fun.

They are also a powerful simulation tool. When you roleplay the vendor-side, you're stepping out of your normal headspace, and attempting to imagine the system that you are part of from another perspective. In terms of evaluation, this is a critical skill.

When you are roleplaying the State Officer engaging with the vendor, you're developing and practicing questioning skills that are fundamental to your work. But, you're practicing them in a safe environment where, if you make mistakes, you get to step back with colleagues and discuss what might have been a better approach.

The point being: regardless of which role you are in, you're developing a powerful set of skills and habits of mind that are intended to improve your ability to imagine and detect bullshit (Learning Goal #2) as well as improve your confidence in digging into the work of your states and their vendors (Learning Goal #4). And, at the risk of being redundant, let's be honest... roleplays are fun. 

:)

Reflecting in community (30m, small group)

In the full group, you ran the roleplays. Take time after the roleplay to come back and reflect with a partner.

CMS oversees large, complex projects that involve many people, lots of dollars, over a long period of time. The program and project managers on those projects are critical stakeholders in the process; they have leadership roles in the design and delivery of the software, and they are not without hopes, biases, and faults. As a SO, it is up to you to understand not only the projects they are overseeing, but these people and the roles they inhabit, so that you can better work with them and understand the progress being made on delivering excellent tools to the people.

Take some time to reflect on the roleplay, and the thoughts it inspired regarding your work with your state.

  • What questions might you begin asking going forward?
  • What processes do you think you might want change?
  • What processes are working well?
  • Are there more things you feel you need to learn to tackle this space better?

Those questions are offered as starting points only. You are welcome to take your reflection in whatever direction is best for you.

In the guides

This lesson is the beginning of a journey. If you're interested in learning more, there's material in the 18F Derisking Guide that you'll want to check out.

From the Federal Field Guide:

From the State Software Budgeting Handbook:

Wrapup (5m, solo)

Take a few minutes to share your reflections on this lesson.