Attention!
The content on this site is a materials pilot. It represents neither changes to existing policy nor pending new policies. THIS IS NOT OFFICIAL GUIDANCE.
Assuring quality
Procurement flexibility
Ask how the team implements quality monitoring through their contracts.
- Bad: The state has little or no expectations for monitoring quality in their contracts.
- Meh: The state has vague expectations around monitoring quality in their contracts, mostly leaves monitoring to development teams.
- Good: The state can demonstrate how they monitor quality in their contracts and work with teams to make sure monitoring is implmented.
What's this about?
Quality monitoring is an integral part of government contracting. For quality monitoring to be truly effective, it should speak to the behaviors that will lead to a successful delivery. This is a critical step towards acknowledging that software is not a product one buys, but an ongoing service that must be shaped and supported on an ongoing basis.
Lesson outline
There are two things that are important to keep in mind about the assurance of quality in the production of software products. First, in the context of government work contracted out to vendors, the metrics and guidelines for quality should have been baked into the contract. If those are absent, it may be difficult to hold the vendor accountable. Second, quality assurance is a many-dimensional concern, involving technologies, practices, processes, and an overall mindset towards producing something that works and is maintainable long-term.
Active listening: A conversation on QASPs (~1h, solo)
This lesson's material is a conversation between Randy Hart and Matt Jadud. In this 30 minute conversation, we explore what a QASP (Quality Assurance Surveillance Plan) is and why a plan that accounts for change and agility matters in the context of government contracting. A full transcript is (with a few "ums," "ahs," and flubs removed) is provided below, if you'd rather read the conversation.
A funky conversation about quality assurance. Transcript below.
Dimensions to a QASP
There are seven dimensions to a basic QASP discussed in this conversation:
- Tested code
- Properly styled code
- Accessibility
- Deployed code
- Documented code
- Security
- User research
In the conversation, these are called out pretty clearly.
Listen actively
Listen or read this conversation actively:
- Keep a thread running in the back of your mind regarding your own projects, and turn up your “bullshit filter.”
- Pause the conversation at any point that you hear something that makes you think about some aspect of a project you’re working with that makes you… wonder.
- Reflect and take notes on the themes and ideas discussed to share with your group. In an active listening/reading process, it is this reflection, not the content itself, that is most valuable.
Some possible examples:
- A feature that has taken forever to come into reality.
- A criterion for delivery that you thought was impossible, or even easy to achieve without delivering value.
- How a better QASP would have avoided a particular problem that you’re seeing.
- How your own projects are working similarly to this, and doing well (We don’t always have to be critical, after all.)
The process of listening actively takes time. You can listen closely on a first pass or have a first listen on a walk, then sit down and do a second listen where you take some notes. Do what works best for you.
A conversation on QASPs
| Matt | Good morning. This is Matt Jadud, I’m an Innovation Specialist at 18F, and the topic today is QASPS… or is it “KASPS?” I mean, how do you pronounce that? It’s Q-A-S-P. So, I’ve asked my colleague Randy Hart to share his experience with costs or “quality assessment surveillance plans,” and their role in the procurement and contracting process. So, Randy, could you tell us a little bit about who you are and what you do at 18F? |
| Randy | Sure. I, my name is Randy Hart. I’m a former contracting officer, and I’ve been with the federal government since 2002. I joined 18F about five years ago, after having lived through a number of nightmare contracts that I was the contracting officer for. And I just wanted to see if there’s a different way of approaching contracting, when it comes to it and digital services for the government. And so for the last five years, we’ve been tinkering with ways of approaching contracting differently. Holding contractors accountable for delivery versus documentation. And it’s been a lot of fun. And I’ll say it’s its quality assurance surveillance plan. |
| Matt | What did I say? assessment? I said, assessment? Well, we’re not gonna be able to edit that out. So: quality assurance. It’s noted for the future, and it’s recorded in the past! I spent a short time in defense contracting, and I think part of what I’ve started to come to see in reading about and working in this space from the engineering side, as an engineer at 18F, is how this really can become more of a dialogue and a collaboration with the contracting officer. So I’m personally I’m excited about it from so many different dimensions. As part of your work at 18F you’ve collaborated Robin Carnahan and Waldo Jaquith on the State Field Guide. Could you say a bit about that document and we’ll, we’ll throw a URL somewhere so that people listening can look it up. |
| Randy | Sure! That document was all about trying to capture what we’ve been learning over the last four, five, six years at 18F, and about different approaches to budgeting to contracting to oversight when it comes to these big it contracts. The main focus of that was to capture what we’ve learned as a team as we’ve worked with different federal and state agencies, to capture the lessons we’ve learned and some of the things that we think should be focused on by folks that are procuring IT systems. |
| Matt | So, I am new to 18F work. Actually, I’ve been told I can’t say that anymore… I’m approaching six months with 18F. We’ve released the The 18F De-risking Guide, which is a document that really brings together a lot of expertise from across 18F and captures years of experience there, as well as from people we sometimes refer to as friends of the show. And, that document explores how we can develop better software solutions at the state and federal level. As I was reading through this, one of those things is the QASP, or the quality assurance surveillance plan. So what in your words is a QASP, and why does it matter? |
| Randy | Yeah, so what we typically do is we’ll write a statement of objectives of where the vision is for what a product needs to do. It talks about user research and the things that we expect the contractors to do as part of delivering software. The quality assurance surveillance plan is something that’s required in federal contracts to talk about how you’re going to oversee the work being done and ensure quality. In the past, they’ve been really focused on a lot of documentation, things like program management plans, schedules, and just a whole lot of written artifacts. And what we’re trying to do with the QASP that we’re using in our contracts is focused on “what is good software?” What does good software look like? So we focus on delivery, and actual, good working software. How often should it be delivered? What are the good elements of a proper, properly working software development team? |
| Matt | So, part of how I think about it… I spent years when I was younger, don’t ask me to play now… but I spent years playing the piano. And I sometimes think about this almost from a performance perspective. If you think about the software as a recital, right, the service that you’re delivering, you’re not going to get there by writing about the act of practicing the piano. It’s actually going to be by sitting down with the keyboard every day and practicing. Therefore, as we explore this, my read of the QASP is that it really speaks to the day-to-day practices that lead to excellence, as opposed to a framework where writing about what you’re doing might, you hope, get you there. |
| Randy | I just want to say unlike my previous experience with IT contracts, I could ask for all the plans in the world, I could ask for all the written documentation in the world… and things would look like they were going fine, because they were writing about things that might or might not have been happening… but I wasn’t seeing actual software. The mission areas that I was buying on behalf of couldn’t show actual progress; they just wrote about progress. And so this is the shift we’re making: from memos to demos. The most important thing is not getting just a bunch of memos about what’s going on, but demonstrations of actual work and software. And that’s what we try to focus on, as, as we wrote the cost, and we get rated on it. |
| Matt | And so the QASP in the De-risking Guide, thinking about moving from writing about (or memos) to actually having working software speaks to seven different things. And there could be more, I’m sure. But we talked about tested code, properly styled code, accessibility, deployed code, documented code, the security of the software service that we’re producing, and user research. So we’ve got those seven things. Of those seven, are there two that you’d want to highlight first? Are there any that sort of stand out to you over your years of experience that think are particularly critical? |
| Randy | Yeah, to me the deployed code and documented code are probably the two most important sections of the QASP, just because, in my previous experience, I just wasn’t getting that. As the IT contractors were going about their work, I wasn’t getting code that was actually deployed or ready to be deployed. I wasn’t getting documented code. So I was in a black box, I didn’t know what was being done. So the deployed code is all about showing every two weeks that there’s something that can be deployed, it’s being iterated on, it’s being built on based on good functionality that’s backed up by user research… but also documented. One of the big things we talked about in the De-risking Guide is the term we call vendor lock in. When you’re completely, bound to a single contractor, and they own all of the code and all of the documentation. It’s very tough to break off that relationship and have another vendor build on top of what’s being built. And that can be intentional… it’s in the vendor’s interest to own the code and have it in their own environment, because then you’re dependent on them. |
| Matt | You you don’t have a marketplace anymore. And I know I had colleagues who would talk about building an empire, right? That is to say that if you get in early on a project, and you then succeed through subsequent phases, that eventually you get to a point where there’s a long term relationship that gets bought into on the government side… and you potentially then hold the keys to critical systems. |
| Randy | The most negative spin on this is basically the government’s held hostage if they don’t have documentation of the code. And they’re held hostage by companies that may or may not be doing it intentionally. And that’s, that’s the paradigm we’re trying to break out. And so that’s having documented code that the government can see, is so important to break out of that. What do you see as the most important parts of a QASP when you read through it? |
| Matt | I don’t know what the most important are… I mean, this is this is a relatively new space for me. I think two, maybe even three, because I’m going to have a hard time separating out two of them. One is user research. So… as as an engineer, when I’m on a team, I do not like getting out ahead of my UI/UX team members… user interface/user research. (And I have to admit, there may be new words and terminology in this space that that my colleagues might use to describe their work.) But essentially, I don’t like writing code that’s not going to serve the user. I really value deeply my colleagues who are working with users to understand needs to capture all the complexity when we’re talking about things like unemployment benefits systems; there’s a lot of reality for people who live in the United States as they engage with those systems. How do we then build systems and services that meet the needs of people where they are? I think that the the user research piece is really critical. |
| Randy | And I just want to add in there, this isn’t related to the QASP, but that’s why we don’t spec everything out in our piece. You don’t know at the time of contract being awarded what all of the requirements are of the users. You don’t know from headquarters, you can’t sit in a room and say “this is the system and it shall do this.” “The system shall do that.” It’s about what direction you need the system or the product to go, and let the user research kind of build/help you build the product in a way that’s going to benefit the actual users of the system versus headquarters spec’ing everything out at the time of the RFP. |
| Matt | I think that’s just essential. And in that regard, I think I’d almost tie that together with a commitment early to accessibility. And we’ve got to the USWDS, the, excuse me, the United States web development system—its acronym soup—(Randy: it’s Web Design System)—I’ve started using that on projects here at 18F. And it’s just marvelous to have a framework that, as I’m developing a web based application, makes it easier for me to build tools that I know will be accessible to the broadest possible audience. Because it’s really, I think, too easy (too often) for engineers to get focused down on details, and lose sight of the fact that we are building software for people who come from many places with many abilities. When the government builds tools, they need to be available to everyone. So that user research and accessibility piece, I think there’s there’s a tight coupling. |
| Randy | Just to build on that from my experience with federal contracts. There is something called Section 508; that’s what sets out the standards for the Americans with Disability Act, and we would include that in our contracts. But there wasn’t enough actual teeth in it to say “you are going to build something that’s accessible.” And so what we try to do with the standard we the WACAG guidelines—Web Content Accessibility Guidelines—which attempt to put into place standards for vendors to comply with five away. And it’s it’s a little bit more detailed than the broader Section 508 standards. Holding vendors to that standard is really important. |
| Matt | I think the last piece that I’d mentioned there is the notion of testing. My view, as we’re developing these kind of systems, is the idea that they are tested from the beginning, which to me is a way of an engineer expressing their understanding of how the service or piece of software should work. There are a lot of different views on testing—when and how. But fundamentally, for me, an engineer who is able to think through how a system should work, express that in code as a test, and then develop the system is demonstrating a deep understanding of not just the technical issues, but—when done in hand in hand with good user research—is starting to deeply understand the needs of the end user. And so testing, to me is really a fundamental expression of understanding what you’re delivering to, to the end user. |
| Randy | Just to riff on that a little bit… my experience in the past, before 18F happened—and I’m not an engineer or anything, I’m just contracting person—but my experience was testing happened way late in the cycle. People would wait and not write tests as they developed the software, and then they would hand it to the testing team, and testing team would write some tests… and then things would break. And, like we mentioned earlier about not having well documented code, when the test would break way at the end, we would cycle all the way back, and nobody could figure out where things broke down. What we talk about in the QASP is: every two weeks—every sprint cycle—the tests should thought about in the beginning of the sprint, and then run at the end of that sprint, and it should be a continuous process. And that’s very different than what I used to experience with my IT contracts. |
| Matt | There’s a lot of ways to go about testing, and I’m fairly confident that there’s a lot of people with very strong opinions about it. But no matter who you are, certainly, if your delivery cycle is on the six-month-to-a-year-timeframe, and you’re only testing at the end of those six months, or at the end of the year, ur doing it wrong (to quote the the cool kids on the internet). So, we’ve covered a bunch. We’ve covered six of these seven, right? We’ve talked through most of these, but we haven’t talked about security. So what does the QASP say? And why? Why do we roll security into the QASP? And where do you think why does that play a role critical role, do you think? |
| Randy | It puts on notice from the beginning the fact that security should be baked into the work being done. What you don’t want to have is not having secure features being built, you don’t want vulnerabilities when it comes to the code being built, you don’t want any exposure of PII (personally identifiable information) or anything else. You need to make sure that the contractor you’re working with is building things in a secure way. In my previous experience I had contracts with companies that were building on top of some kind of COTS software that had vulnerabilities, and our security team would identify those things and say “Hey, this is this is not a good thing, you need the contractor to fix this bug. It’s, it’s vulnerable.” I had no way of enforcing it, and this says we’ll have our security team run tests and vulnerability scans from the beginning to make sure that what’s being built isn’t vulnerable. And so that that’s why we put that in the QASP is just to make sure that the vendor team and the software developers are aware that this is a an important thing, and it’s not something they can just write a document about; it’s actual practices for secure, secure development. |
| Matt | I’ve been on projects doing code review, where—it must be? Well, it was no, I think it was in our QASP—we had a very, very healthy relationship with the vendor, so this wasn’t difficult—but as code was coming in, and we were doing review, there were things that I saw that it’s like, you can’t do that. Not not from a “it’s illegal” or “it’s morally reprehensible” perspective, but they’d be doing things that technically is just like “if you leave that there for any length of time and you start to build on it, you will be leaving gaping holes in your software.” Being able to go back to the QASP and point to security guidelines and say “the guidelines that we’re using here, three dot one, three, dot two and three dot three all speak to exactly what you’ve just done.” |
| Randy | And the best contract is one you don’t have to look at after. If you have a healthy relationship with a vendor, in the best case, they’re doing things the right way, but it is nice to have the QASP if things just aren’t going the right way. And it sets some high-level guidelines, not drilling down on specifics about every single thing that a healthy software team should be doing, but it’s at least there and sets the threshold to help, you know, collaboration between the government and more contractors. |
| Matt | And in this case, as I said, it was a good healthy relationship, but it was nice to have that as a sort of as a backstop. It was a reminder, like “you’re doing all these good things that we talked about in the QASP, but don’t forget this one, too.” “Aah, you’re right, our bad.” And so it was nice to be able to have that to rely on early in the process. |
| Randy | I think the two things that we’ve found that we’ve had to make adjustments to as we worked with vendors that we worked with is, is the security piece—making sure that they do build build in security from the beginning and don’t cut corners. And the accessibility piece that you mentioned earlier. It’s something where software developers can try to cut corners or not meet the intent of the QASP. And that’s something that as vendors build things that you, as the buyer, need to be aware of. |
| Matt | So we’ve, we’ve talked through all seven of those [the QASP dimensions]. I have a closing question or space for a little bit more conversation. You know, I could be sitting here and say, “This is really cool. And I want to work some of these ideas into into a future contract.” And depending on who you are, you might even be able to reach out to 18F and talk to people like yourself about that process. But what if I’m an agency or a state already in a long or large contract with a vendor? And I’m realizing there are things in this conversation that I wish I could start to bring to the table? Is there anything that a state can or an agency can say or do? Or do they just have to wait? This is perhaps a question demonstrating my lack of understanding of the contracting process or actually, and by using contracting and procurement interchangeably in a way that I shouldn’t? Well, that’s two questions. So… |
| Randy | The challenge here is that there’s metrics—and we talked about in in the field guides—if a contract is over $10 million and over three years long, it’s got a 93% chance of not delivering either on schedule or on functionality or on cost. Recognizing that’s the environment right now, with most government contracts, and recognizing the most states… they don’t want things not to go well! Nobody wants things to go sideways! But the paradigm has been outsourcing way too much without having product ownership. The states have to claw back ownership of the products that are being built for their end users. And so, recognizing that’s the environment, recognizing what the private industry has recognized probably 10-20 years ago, that there’s too much being outsourced of their missions, and that same thing is happening in government. And so it’s not going to be a quick fix or an easy fix. Most states are in long term contracts, that that about source way too much and specked out way too much in their therapies. So, I think having that understanding and knowing that it’s going to take time to turn the needle back is important. A lot of the data that the states should be owning is completely within contractor systems, and it’s a black box. So, it’s going to take time to transition to a better model. But I think it’s it’s critical that the way that we approach this is to understand the environment as it stands right now, and look for ways to find vendors that don’t have that predatory attitude when it comes to delivering software for states. It’s a it’s a huge, it’s a large market, and there’s a lot of money involved… go ahead! |
| Matt | Something I feel that I’ve I’ve started to believe, being relatively new to this space, is that it all feels to me like something that creates a healthier marketplace. By which I mean: if you have these practices baked into your contracting process, if you have a commitment to well documented systems, if you have a commitment to the states owning the data… then at some level, when you commit to the state owning the data, and the state owning the code, and the state owning the process, it means the vendors then really are starting to compete on their ability to deliver excellence. If you are thinking of it as a relationship building process, we could still end up in a space where you have a vendor who has a long term relationship with a state. But instead of it being a long term relationship because this state has no idea how they would get their data out of some system, it’s because there’s a vendor who is consistently delivering excellent, well tested, properly styled, accessible, constantly deployable, documented, secure code that’s backed by excellent user research that is constantly being tested in the hands of the people. |
| Randy | Yeah. And, this is probably another conversation, but we try to write RFPs or solicitations in a way that invites those sorts of vendors—at my last agency, one of my RFPs was like 400 pages, and nobody read it. We asked for binders and binders of proposals and nobody could read through all that. And it was all this idea that you can get away from risk by writing down more things. But at the end of the day, if things aren’t being delivered, there’s not a whole lot you can do. So what we are trying to do with our solicitations is reduce all the noise and make the RFPs accessible to vendors that might be excellent, but maybe not in the government space—but want to be part of the government space—or trying to increase the market, for those kind of vendors, most of which are small businesses. They’re really strong, small businesses that can do this work really well. But, they read these solicitations and they’re just terrified. You know, they’ve got all these codes and all this stuff that’s just not necessary, so with the QASP kind of laying things out, what we see is excellent delivery. And then there’s a vision statement and objectives written in the RFP that are less prescriptive and more about where we want to go with the products being built. That attracts vendors that are really good at this work, and that’s really what we try to do at 18F with our acquisitions is lower the barrier of entry for vendors that might want to be part of the world of civic tech. And that’s just that’s one of our focuses as we as we work with different agencies. |
| Matt | I think that’s a great spot to end. Randy, I want to say thank you. We recorded this on a Thursday, October 29th, 2020. You’re coming to us live from (Randy: Fredericksburg, Virginia). And, to hear you tell it, it’s one of the best places on earth. I’m recording here out of Lewiston, Maine. Again, thank you so much. |
| Randy | I enjoyed it. |
Discuss in community (1h, group)
You will need someone to volunteer to take some notes. Whomever was born after (but closest to) January 20th should be the note taker today.
- Check in. (5m timer) While people are arriving, check in with each-other. How is everyone doing? Take a moment to share something positive from the week, either at work or at home.
- Centering. (3m timer) We jump from meeting-to-meeting and there’s nothing healthy about that. You will get more from the next hour if you’re here. A simple breathing exercise (breathe in on 4, hold 4, exhale 4, hold 4) is a good way to clear your mind and body. There’s lots of resources online (4m20s) regarding simple centering exercises that you could investigate and use at the start of group conversation.
- Focus. (1m timer) Take one minute to identify one or two insights this conversation led you to. Make a note or two in your notebook so you can be focused when you share out.
- The seven dimensions. (30m timer) As a group, first share out which dimension(s) you found to be most interesting in your reflections. Then, after you share out which dimensions of the QASP inspired thought, go back around and take a minute or two each (round-robin) to share why that dimension triggered insight. This should take roughly 30 minutes total, and try and create space for everyone to share out.
- Transformation. (15m timer) Were there themes that you saw emerge from your insights? Commonalities across projects? Identify what you saw as a group. Then (and more importantly), do you have any thoughts about your process with states, and how you might transform your process so as to improve outcomes? The note taker should try and capture the group’s thoughts regarding themes and process transformation for sharing back out to the group/community.
In the guides
This lesson is the beginning of a journey. If you're interested in learning more, there's material in the 18F Derisking Guide that you'll want to check out.
From the Federal Field Guide:
From the State Software Budgeting Handbook:
Wrapup (5m, solo)
Take a few minutes to share your reflections on this lesson.