Using Exploratory Stories to Stay Sane

While it can be tempting to dive right in and start coding as soon as a new story gets assigned to you, it’s often unwise to do so. When operating in an agile environment, we normally need to estimate the level of effort a story will require. If you’re going to commit to saying that a story will take you three days, you need to do whatever you can to stay on the right side of that estimate. You should always be under-promising and over-delivering, spending as little time as possible with your stories festering in the “in progress” column.

While you may want to angle for low-pointed stories in an effort to showcase your capability and efficiency,  I’ve found hands-on Spike stories followed by Proofs of Concepts (PoCs) to be essential tools for not only accurate estimates, but ensuring that sufficient time is spent designing a reusable approach.

In addition to proving useful for engineers within the flow of a sprint (or a Kanban… epoch?), the Spike/PoC combo can prove useful in epic planning. Having an estimate for upcoming work - no matter the scale - is infinitely better than a shot in the dark.

What is a Spike?

While one might assume SPIKE is some nebulous acronym from the floppy disk era of computing, this isn’t true in the Agile context.¹ The term derives from rock climbing, where spikes are driven into the rock face so climbers can hook in their ropes, giving them a checkpoint that they can hang from if they fall. Preserving the climber’s progress on the wall also ensures that they won’t fall to their death in the event of a misstep.

The rationale behind the metaphor to Agile is that while hammering in a spike doesn’t help your velocity (you’re hammering, not climbing) they ensure the stability of your current position and improve the velocity of future climbers. It is inherently a forward-looking concept.

In its initial use², the term was more akin to a proof of concept than a research task. A Spike is a time-boxed effort to produce the simplest possible solution to the problem at hand. In the modern workplace, the time-boxed nature of spike work remains, but the term generally refers to exploratory research into the given user story (or bug, etc). 

Too often, engineers think that working on a research story means that they won’t be writing any code, just digging through the codebase, reading documentation, or learning more about the services being used. While a Spike story will almost certainly include some or all of these activities, sometimes there’s no better or more efficient way to learn than by getting your hands dirty with code

Once you’re ready to close out the Spike, you should update the story with a description of the work you did and what you learned, then share those findings with another engineer before closing out the story. This helps ensure that the knowledge gained from the story doesn’t disappear with you if you get sick, are otherwise unavailable, or if the follow-up work gets deprioritized. It’s good to find moments where you can keep your teammates up-to-date on what you’re working on with digestible information. This can prove immensely helpful later on. 

What is a Proof Of Concept?

Question: “If your Spike work includes coding, then how does that differ from a proof of concept?” 

A Proof of Concept (PoC) tests out the desired pattern for the solution with a limited blast radius, Spike coding just encompasses writing the bare minimum new code needed to indicate the potential of the desired functionality. 

Given the time-boxed nature of a Spike story – generally one or two days – you want to do the minimum research required to code a quick solution that indicates whether your intended approach will work. Nothing unearths unforeseen complexity and unintended side effects like testing. As mentioned before, this simple solution does not need to adhere to best practices or any efficiency measures. Therefore, none of the code used in a Spike should be reused. It’s not the first pass at coding a solution, it’s just a test. 

The first attempt should come as part of your PoC. If necessary, you can presage the PoC story with a design review story in order to spend more time working out an approach to the problem that does adhere to best practices, and run that approach by other engineers / your architect if need be. 

The PoC work, once you start it, will be captured by a user or bug story. It should encompass a strategically selected vertical slice of the application(s) and designed in such a way that it can be scaled horizontally³. 

While this first proof of concept often shouldn’t be immediately accessible to users, you’re writing production-quality code, meaning that the solution you’ve settled on should be testable, extensible, and compliant with your team’s agreed-upon best practices. 

It’s entirely possible that something goes awry during your proof of concept. If this does occur, don’t despair. Again, it’s a proof of concept in more than name. Given all the research and documentation you’ve done leading up to this point, you’ll have empowered your teammates to be up to speed on your approach and you can pull them in for a second opinion. Given that your story was for a proof of concept, you can close that story out, open up a new Spike or architecture review if necessary, then embark on a new proof of concept. 

If your PoC goes according to plan, it may be possible to release the fix/feature to prod and then continue to work on extending it to horizontally cover the rest of your application(s), weaning your code off of the old pattern and onto the new, delivering to your customers iteratively in order to strike a balance between code quality and time to market. If the feature at hand can’t be released iteratively, you can still close out the PoC story and pull in a new story/bug to complete the feature/fix. Given all you know at this point, this story should be easy to point appropriately. 

Breaking down your efforts using exploratory stories can really have a very powerful effect on the way you think and work. Their chief benefits can be broken down as follows. 

Benefit 1: Better Intra-Team Communication

‘Exploratory stories’ could also be thought of as ‘explanatory stories’. You’re likely going to be working in a pod with product people, tech leads, and designers in addition to fellow engineers. Your non-coding coworkers are going to understand very little of the technical details you share in stand ups and, to be honest, it’s also a lot to ask your fellow engineers to stay engaged with the problems you’re working on based on the breadcrumbs shared in meetings like that. 

In order to keep our teams up to speed, it’s important to leverage our shared language of agile. Everyone on the team knows, or has the capacity to learn, what a Spike, Proof of Concept, and Design Review is. Everyone also has the ability to understand the functional utility of going through the necessary steps before leaping into coding a solution.

Benefit 2: Improved Estimations

Insidiously, even if efforts (like adopting fibonacci or some other abstracted ‘pointing’ strategy) are taken to base story estimates on something other than time, a new shared vocabulary for how long stories of a certain size remain on the board begins to take form. 

While you don’t want to commit yourself to an overly-optimistic timeline, you also don’t want to look like you’re milking what other engineers or products consider to be an ‘easy’ story in order to luxuriate in your task. This aversion often leads new (and some seasoned) engineers to underpoint their stories.

My team practices blind pointing⁴ and writing stories in such a way that they could be picked up by any engineer on the team to help offset this phenomenon, but by itself that doesn’t solve the issue of engineers getting stuck with an unexpectedly complicated task. 

An inherent issue with estimation is that if you don’t know all the work that will go into the story, your estimation must be a guess⁵. So it really behooves the team to pull in a time-boxed Spike to better understand the story’s scope and ensure they’re not trapping an engineer in a thick jungle of code that they’ll have to hack their way out of. 

Benefit 3: Good Code Stewardship

That brings us to arguably the most important benefit of using exploratory stories.

An engineer who’s behind on a deadline is a threat to a codebase. In the best case scenario, the delayed engineer will normally wind up repurposing existing solutions or crafting something a little too case-specific, then logging a bunch of tech debt items for later improvement. 

In the worst case scenario, missed deadlines result in the perpetuation (and probable further degradation) of already awful patterns, with no regard for best practices, efficiencies, or testability. 

As you take on new work, you should always feel like you have enough time to not just tackle your problem the right way, but leave everything you touch better than you found it. While there’s never enough time to perfect every part of your code base, when you break stories down into planning and research tasks, you can have team-level discussions about the current state of things and what needs to change before you can move forward responsibly. 

Operating with transparency and sharing concerns amongst your team with leadership present gives you a lot more leverage to advocate for constructive redesign then when you’re behind schedule. Effectively documenting and sharing your Spike findings also allows your teammates to weigh in on and advocate for well-designed solutions to the problems you unearth.

Conversely, not effectively sharing the issues you’re struggling with and siloing yourself can turn a good team into an antagonistic entity. Rather than serving as other minds to bounce ideas off of, your teammates’ perspectives begin to seem threatening, ready to poke holes in your approach and call out additional work you need to pull in. Your product owner goes from a helpful conduit between engineers and their customers to an exacting taskmaster and your manager, wondering why this simple story is taking you so long, might put guidelines on the work you take on in the future.

We’re living in a professional landscape dominated by agile and in order to be good stewards of our codebases (and our psyches), we need to know how to efficiently navigate the shared vocabulary it provides. 

This is an abridged version of a longer blog post on the same topic. For the full version, which includes a playful example demonstrating the tangible benefits of using exploratory stories (and the risks of not using them) click here.

¹ SPIKES is an acronym used in nursing for deliver ‘bad’ news to patients (scare quotes to emphasize understatement)

² As far as I can tell, the Spike story was first defined in Extreme Programming (XP)

³  A “vertical slice” of an application encompasses an entire workflow as it traverses the front end, your business logic, your database, and then back to the front end. The slice covers all parts of the application, from “top” to “bottom”, however it’s “narrow” in that it doesn’t have to cover all the possible request paths, just one of them. You can then scale the solution designed using a vertical slice of the application “horizontally” by covering the rest of the potential request paths. 

⁴ When writing the story, we leave the points section empty. When we refine the story, someone reads it out, then the PO opens up a poll and everyone votes on how many points they think it’s worth. If the result isn’t unanimous, we then have a discussion about why we chose what we did and decide on what we think makes the most sense, normally defaulting to the highest estimation, unless it was based upon an incorrect assumption. A lot of previously unrealized complexity is called out during this process, as everyone has a slightly different frame of reference for the problem at hand. 

⁵ Which fact, among many others, calls the whole exercise of estimation into question #noestimates