We can routinely deliver value by understanding customer needs and satisfying them in small increments. Here we suggest an approach for work that transcends this process.
We recognize that programming involves some chores that are not known to customers but that will impact the ability to deliver increments if left unattended. This too should be done incrementally and rolled into the velocity used to schedule work.
Less recognized is that whole subsystems can be built from the vague recognition of need and that these can solve problems that only years of systems experience makes known. This work must be undertaken outside normal prioritization.
Tom DeMarco's Slack discusses the danger of too much efficiency and how it might leave developers unprepared for the work ahead.
Clayton Christensen's Innovator's Dilemma explains a similar but more perverse problem at the product management level.
Method
We suggest a role that might be called staff engineering that reports higher in the organization than teams that hold routine commitments.
We assume the role provides visibility and access to strategic decision making. Often engineers at this level are expected to produce reports. Here we consider instead simply taking action rather than suggesting it.
Find something that needs to be done.
Prefer things that are feared too complicated to be successful.
Prefer things with long term payback that might not prioritize well.
Prefer things that a smart person started but abandoned for lack of time.
Develop the thing cautiously, looking for great solutions, avoiding deadlines.
Help other developers take advantage and get credit for their work.
Let go when someone else is ready to take over.
Result
Often a great solution is made great by its non-obvious simplicity. By making this a non-show-off activity one is free to look for short-cuts that are still fit for purpose.
Often success depends on a long sequence of experiments. By working off the iteration cycle experiments can run, data collected, business consequences assessed, and, next steps initiated.
Often the clues to success are buried in the ongoing work of iterations. By not separating exploration into a separate R&D organization, the existing routine progress can inform and redirect staff work.
Often the codes created for staff work can be integrated into development cycles. By keeping solutions small and focused they are more easily understood and ultimately adopted when proven useful.