There is no Step Two, or “What Comes After MVP?”

Posted on by under Business

An incredible amount has been written about Agile and MVPs over the last 20 years, yet the majority focuses on the first step of a process, the output of which is stated to be a minimum viable product. This mismatch is likely a side effect of most literature being focused at organisations transitioning from out-dated practices and adopting something “new”[1]or desperately clinging on to the past in the case of SAFe – I’ll get to that article, eventually. There is much less writing dedicated on what to do when you’re up and running, with the general guidance being to reduce cycle times.

The conflation of process and product is confusing, often resulting with the view that if you simply “do agile” then the product will take care of itself. Often an MVP is the conclusion, with minimal discussion on what an MVP is outside of an expansion of the initialism. Let me give you an example.

About 5 years ago I was involved in a pitch to an academic publisher who wanted to completely rebuild their website. They made it clear in the tender that they did agile, and that they wanted to release the MVP alongside the celebration of the company’s birthday which kicked off on New Year’s Day.

My section in this pitch was to discuss this “MVP”, and I took the opportunity to lay some ground work.

  1. What does MVP mean to your business?
  2. What can you do without, or copy from the existing site (e.g. content)?
  3. Given your 1st January deadline, when do you actually expect to release the new site?
  4. What are you looking to achieve?

The answers were what you’d expect. MVP = Minimum Viable Product. Everything was to be new (code and content). To release a) when the fireworks went off at 00:00 on New Years Day, or b) over Christmas, as long as everything is finished. The aim of the project was to do the project, but any further “why” was elusive with no cohesive view shared, not even metrics.

Rather than doing agile, they had changed their terminology and kept everything else the same. MVP was the new word for a release, the solution was predefined (a complete lift-and-shift), and the scope was locked in. It’s waterfall, plain and simple.

In this instance there was no MVP, there was a project delivery. Even if delivery was managed through sprints, and some tweaks made based on stakeholder feedback, it still won’t have been an “MVP”.

Why? Because an MVP requires an hypothesis to test.

The purpose of an MVP is to find out if you’re right, as quickly and cheaply as possible. If there is no hypothesis, there is no product to access the minimum viability of.

I joined Safecall because they had a particular strategy they wanted to validate, and needed a Product Manager to shepherd it in[2]In the interest of full disclosure, this was started before I joined the business. I turned up towards the end of the initial delivery from an external supplier, but had been in post nearly 6 months … Continue reading. Would companies buy software from an expert provider with 20 years experience in whistleblowing services, but very little in building software?

Given the company’s lack of experience in the area, the last thing anyone would do is to write a cheque for several million quid to build a whistleblowing platform from scratch. As such, we used pre-existing products built to handle entities related to individuals within businesses to build our MVP. I also took the step early on to purposefully keep every page and interaction as simple as possible. Simple in this context means few screens, little conditional logic, and little fluff. For the first release, for the vast majority of Safecall clients, the platform operated as a secure means to receive and transfer confidential information, and little else.

We gathered feedback from existing clients who had been granted access to this new system, as well as through the sales process. The most common feedback was that “it’s a bit simple” – considering it isn’t “what are you doing, this is rubbish” I take this as a pretty big win. Over the last 2 years, the platform has become a core part of the service we provide and we have fleshed out some features, added some new ones based on new hypotheses, and reworked some areas where our previous assumptions were proven wrong, or we discovered unforeseen consequences. Ultimately, what we’ve proven is that, yes, companies will buy software from industry experts, and that it isn’t only “software companies” that can play this game.

An MVP is two things, it is a product, and a method of testing an hypothesis. In this sense MVPs are unlike any other part of the empirical, iterative nature of doing agile as it isn’t just a piece of research, it is also the thing you sell. In this sense, they are the most expensive way to ever prove whether you’re right or wrong.

The answer to “what comes after MVP?” is another hypothesis, one based off of the new data you have now been able to collect. This new hypothesis and new MVP can take many forms, from scamps to research aides to A/B tests, but your aim should always be to do just enough to clarify your assumptions, and test your hypothesis. Or, to put it another way:

Simplicity–the art of maximizing the amount of work not done–is essential.

https://agilemanifesto.org/principles.html


And then you do it again.

Notes

Notes
↑1 or desperately clinging on to the past in the case of SAFe – I’ll get to that article, eventually
↑2 In the interest of full disclosure, this was started before I joined the business. I turned up towards the end of the initial delivery from an external supplier, but had been in post nearly 6 months by the time we’d made our first release in March 2019

Kill your product

Posted on by under Business

Last time, I ended by summarising that to progress you need to quickly iterate, and threw in a link to the wiki page on Fail Fast. A couple of posts ago I was waxing lyrical about bass players, and ended saying “Print that 7″, and move on.“. The theme here is obvious:

  • Do something
  • Pay attention to what happens
  • Use your new knowledge to do the next thing

When speaking about doing agile, especially when migrating away from outdated methodologies, people often focus on the individual side of it, the most obvious being differences between Command-and-Control Project Management versus Servant-Leader facilitators.[1] Scrum Masters, Agile Coaches, etc

But from a Product perspective, hubris is often king. We all know Product people who think they are the second coming, are absolutely indispensable, and have some sort of perfectly unique take.[2]Ed: You really want to open that door, huh? When you consider that an agile approach to Product requires you to run experiments and accept that you’ll make mistakes in order to learn, this is in direct conflict with corporate cultures which reward (perceived) perfectionism by way of arrogance, bravado, and the safe space afforded by feature-and-revenue led project plans. This can lead to people who have been moved in to Product positions[3]Often from previous “decision making” roles, such as a member of SLT, Programme Managers, Project Sponsors, etc as part of a transformation piece deciding that while everything else is changing, they don’t need to.[4]I’ll cover this in greater depth in a future post, for now let’s put a pin in this and move on, but do keep it in mind.

The other part is the wider side – what are others doing?

If you consider a market of similar software products, there will be multiple companies and competitors, each making changes and learning from successes and failures. While no one has perfect knowledge, other people making mistakes for you is extremely cost-effective. And much like how you will look to take advantage of your competitions’ weaknesses, they will do the same, as their aim is to take money out of your pocket and put it in theirs.

If you’re able to learn from the mistakes of others, as well as your own, and act on them quickly, then you’ll have a better chance at staying ahead.

To fully learn from these collective mistakes, Product needs to do something particularly painful, they need to go out with the intention of killing their product. They must view their creation through the lens of a competitor, and rather than make excuses, they must pick it apart, highlight its flaws, and ruthlessly remove them.

You need to get over your own bullshit, because if you don’t kill your product, someone will gladly do it for you.

Notes

Notes
↑1 Scrum Masters, Agile Coaches, etc
↑2 Ed: You really want to open that door, huh?
↑3 Often from previous “decision making” roles, such as a member of SLT, Programme Managers, Project Sponsors, etc
↑4 I’ll cover this in greater depth in a future post, for now let’s put a pin in this and move on, but do keep it in mind.

Time to live

Posted on by under Business

Businesses who conduct any form of software development are increasingly “doing agile“. Typically this means they’ve replaced traditional waterfall project management within their “IT department” to probably Scrum or SAFe[1]For the sake of brevity I’m skipping over why SAFe is a step backwards and only entrenches problems. That’ll need a post all to itself., within smaller and larger orgs respectively.

A few people will go on a course, and on return set up a Jira / VSTS board, book a morning meeting titled “Stand Up”, and the release schedule is changed from Quarterly to Fortnightly overnight. On the surface, this is a step in the right direction[2]It isn’t..

If agility is a concept restricted to just your technical teams, and not your business as a whole, then you’re not actually “doing agile”.

A conversation I routinely have with friends, (ex)colleagues, and the like, is about how tired they are because of the level of extra effort required to affect change within their organisations. This is across all shapes and sizes, big/small, paid/voluntary, public/private.

The conversation will eventually meander through two topics:

  • How long it takes to make a change – start to finish? From concept, to delivery, not just development.
  • How much resistance occurs along the way? Do they feel that it is their own personal determination that makes change happen, instead of something which is embraced?

Complaining about work and feeling helpless isn’t new, and given that it is the plot for three of the biggest films of 1999 I’m not pretending like this is some sort of revelation. However, the frequency of answers[3]My sample is self-selecting because this conversation doesn’t come up in relation to organisations where the answers are “no time at all”, and “very little” being “a long time, could easily be a year or more“, and, “oh, so so much, I’ve given up more times than I’ve failed, let alone succeeded” coupled with the clear frustration expressed is something which leads me to believe that while this isn’t by any means rare, or novel, it’s actually an accepted, expected and purposefully constructed scenario by the organisation itself.

This is often done under the presumption that enough approvals and controls will ensure certainty of two things:

  1. Nothing bad will happen
  2. Only good things will happen

A fundamental aspect of doing agile is embracing uncertainty. To embrace uncertainty you need to be able to make changes quickly, and learn from them.

To make changes quickly, individuals need to be empowered with authority and responsibility delegated. They must be able to make meaningful change without arbitrary approvals and sign off. Equally, they must be responsible for the impact of those changes. That responsibility must be against quantitative targets, not subjective opinions. If they fail, they fail – now learn from it, and try again.

If your organisation can only do things quickly when the house is on fire, and you use executive authority to bypass red tape, then you have systemic issues which need to be addressed. Because if they aren’t, then you better be absolutely right in every decision you make, otherwise you’ll lose to a competitor who can move faster than you and found out the mistake long ago.

It can’t be left up to the determination of individuals.

Notes

Notes
↑1 For the sake of brevity I’m skipping over why SAFe is a step backwards and only entrenches problems. That’ll need a post all to itself.
↑2 It isn’t.
↑3 My sample is self-selecting because this conversation doesn’t come up in relation to organisations where the answers are “no time at all”, and “very little”

Progress

Posted on by under Me

I read somewhere once that you should regularly take a moment to reflect.

To purposefully take time to see where you’ve been, where you are now, and where you’re going. To consider what went well, what didn’t, and how those experiences can be used to improve situations in the future.

To use your new experiences to make better decisions, and to allow yourself to take some pride in the accomplishments won.

The reasoning behind this is clear – if you’re always down in the weeds, you can get lost. Some people will fixate on the negatives, and miss the positives, others will do the opposite. Give yourself space to breathe, and to attempt to understand the full situation.

Within a project, or product, these decisions are quantifiable and should be measured. We can choose to invest in our deployment pipeline to reduce release costs and time. We can refine copy in a marketing campaign to improve conversion.

It gets much more complicated when you try to apply the same process to a more complex situation, like a person.

Within your own professional context – i.e. your career – the process becomes almost existential. Do you continue what you’re doing? Are you happy where you are? Will this get you to where you want to be? Do you even know what that is? Does it matter if you don’t?

Many years ago, an old boss (and mentor) once asked me to imagine a situation where we discovered a fence crossing a road. The job would require us to remove the fence. He asked me what I thought we should do before we removed it.

I focused on the practical aspects of how it would be removed – “We’d need tools to take it apart.” – rather than questioning why the fence was built in the first place. I later found out that this is called Chesterton’s Fence.

Don’t just look at the good and bad – try , too, to understand why we’re here, and why it happened.

Then you can make better decisions on what to do next.

JFDI

Posted on by under Me

Sid Vicious couldn’t play bass, and only recorded parts of a single bass track on Never Mind the Bollocks.

Sandy Miranda can play bass, and in 2006 released a studio album, nine 7″s, three cassettes, and a CD.

Omar Rodríguez-López got bored of playing bass at 12, started playing everything you can imagine, and has written, released, and/or produced more music than I can possibly imagine.

Actually, that award goes to Buckethead.

Even so, the Sex Pistols are generally considered as one of the most influential British bands. Causing the formation of The Damned and The Clash (Paul Simonon also couldn’t play bass, at first) with both bands playing their first gigs as openers for the Sex Pistols. Then, as the apocryphal story goes, a gig at Manchester’s Free Trade Hall on 4th June 1976 had attendees who would end up founding Joy Division, The Fall, and The Smiths.

How does a band with a bassist who can’t play bass have an influence like that?

Or, a better question – how do have the confidence to join, or even start, a band when you can’t play an instrument?

Sid’s snarled expression helps, you just fucking do it. Because who’s stopping you?

A more restrained statement may be that everyone must start somewhere. Edin Blyton didn’t wake up one day and decide that she was going to write 762 (slightly problematic) books. $FamousFootballer wasn’t born being good at SportsBall. The chef’s on in the Michelin Guide didn’t go from microwave mini-pizzas to whatever it is Heston is doing right now I mean come on.

You may have a vague notion or hope that, some day, you’ll fill a room with your creations, but thinking that success is only that – and nothing else will do – will destroy you.

It is a rather trite statement, but you must see how perfect is the enemy of good, and that it’s actually much more important that you do the thing, rather than obsess over it.

That’s how you learnt every skill you have, and that’s how you’ll continue to improve. Oh, and you’re probably not destined for greatness, and that’s ok! Literally no one is! Again, something doesn’t have to be perfect for it to be worthwhile, or the right thing for you to do.

Even Sid Vicious got better overtime, and by the time he died was…alright? I guess. I mean, he wasn’t John Entwistle – but who is?

Don’t let perfection stop you. Don’t spend more energy talking yourself out of doing something than it would take to just do it.

Print that 7″, and move on.

Release, repeat.

Foolish is he who builds his house on an outdated tech stack

Posted on by under Scrapbook

This was originally scribbled, in haste, in April 2015, where it has sat, to be published now without change.

Production works.

Production pays your wage.

Don’t fuck with Production.

In recent months I’ve been noticing that what I thought to be true is not necessarily the case. There is a certain belief within our industry that while ‘new is awesome’ new should also be feared, new is unknown, new is untested, whereas Live is – that – Live. We know what it does, we know its problems and its limitations, and more importantly it’s been proven and is currently responsible for everything that is going on.

Burning it all and starting again is simple when you have no vested interests, when you’re not making money, when you’re still looking for that breakthrough. But, once that has happened, you can’t be scared to re-write, to re-build, to retread old ground, and potentially kill the goose.

We work in an industry that proclaims desires to fail-fast, to pivot, to constantly improve, but is also terrified of bucking the tread due to our continual reliance, nay – obsession, with hockey sticks. We tweak, we change, we iterate, and we lurch based on hunches and hopes, on superstitions and gut-instincts towards what we believe will work, and if we’re fortunate enough to get the right thing at the right time we end up being one of the ten percent that ‘succeed’ we probably didn’t get there by a single plan, an single vision, and a single tech view that took us from launch to exit in 19 months.

Most of us have changed; most of us have had bad ideas; have wasted time, effort, and development resource; we’ve wasted money, hardware, Clouds, and careers on foolish notions and frankly terrible thoughts (I’d in fact put a serious amount of cash on the claim that virtually no one has ever got it right first time) and if, when, we do finally gain traction, we need to – eventually – undo all the errors we made.

A tech stack is exactly that, a stack.

Stacks are inherently disorganised, messy,

Fuck Your Good Idea

Posted on by under Scrapbook

This was originally scribbled, in haste, in May 2015, where it has sat, to be published now without change.

The road to hell is paved with good intentions, or so the saying goes, and never has this been truer than in software development. Everyone knows best, everyone fights their corner, and everyone stops anything actually being achieved. The irony of this post is not lost on me, my intent is something different, I’ll concede judgement to the reader, but I’m likely to fail.

Clarity in vision, with the supporting determination and stubbornness, is what is required to inform people that their idea is not just misinformed, it’s actually detrimental to the continuation of the company.

Everyone champions ‘fail fast’ and in the very next breath caveat its rules, instead of really trying and failing, we continually tinker and delay for months until we have a perfect test and so we have too much invested to allow ourselves to objectively fail. We lose our vision, we lose our sense, we don’t want to watch our baby suffer. We shield ourselves from our possible failure, only to fail even harder.

We are human, we think emotionally – and that is why we are ineffective.

If only we could burn it all and start again.

Baggage is a part of life, it’s a part of love, and it’s part of creation – you will be bias to your own opinions, dismissive of others, and, well, we’ve all heard of “it’s not a bug, it’s a feature”.

Kiva – Year 2 Report

Posted on by under Me

Another year older, another year wiser – or so the saying goes – and while 2014 has been eventual I’m glad that I’ve continued playing with Kiva. I did stop adding new credit about half way through the year, but I have continued to fully re-lend all existing credit, which with repayments of ~$100 means I can fund 4 new loans a month; which is enough to make myself feel good / do something positive (delete as appropriate). When I wrote the Year 1 Report I can remember being surprised at the total value of deposits I’d managed in 2013, so I’m not surprised that re-lending, over continuing to add more credit, became more desirable this last year.

So, what do the figures look like now?

  • Total Loans = 132
    • 2013 = 58
  • Total Deposits = $1,209
    • 2013 = $829
  • Total Lent = $3,450
    • 2013 = $1,600
  • Total Repaid = $2,148.72
    • 2013 = $751.82
  • Number of Fully Repaid Loans = 52
    • 2013 = 10
  • Total Value of Fully Repaid Loans = $1,600
    • 2013 = $375
  • Total Outstanding = $1,148.36
    • 2013 = $797.18
  • Total Amount Lost = $27.92
    • 2013 = $1.00
  • Amount Lost due to Default = $16.23
    • 2013 = $0
  • Amount Lost due to Exchange Rate Changes = $11.69
    • 2013 = $1
  • Countries = 49 / 85
    • 2013 = 45 / 73
  • Total Refunded and Expired = $125.00
    • 2013 = $50
  • Number of Loans Delinquent = 6
    • 2013 = 5
  • Amount In Arrears = $30.09
    • 2013 = $8.96
  • Delinquency Rate = 2.62% (arrears / total outstanding)
    • 2013 = 1.12%

There are a few interesting points, especially around losses, that I’ll take a few seconds to briefly explain.

Firstly, losses caused by the exchange rate fluctuating are something to be expected as part of lending on Kiva – I mention this briefly in the Year 1 Report – and these loans were located in Liberia, Ghana (x2), Columbia, and Mongolia with durations of between 8-20 months, each loss averaging to ~$2.40, or a cheap cup of coffee. I admit that as a percentage of $25 it’s more than you’d like to see, but contrast those 5 losses totalling less than $12 to the $2,148.72 in total repayments and that percentage is no longer a concern, a minuscule 0.5% of the total value of repaid loans since I started this two years ago.

The other loan, the default, is a bit more concerning – but not for the reasons you’d think.

You see, I mentioned a few loans I’d made in the Year 1 Report to give an impression of the different types of loans you are able to make on Kiva, and this defaulted loan just so happens to be one of the those that I listed. It was the one to the watermelon farmer, that lives in Ukraine. Now, Kiva don’t tell you what part of a country someone is located, but occasionally loan descriptions do contain this information, Pavel’s didn’t, but, Agro Capital Management LLC – the Field Partner the loan was made with – do state that the majority of their loans are based in Crimea.

I hope that Putin enjoys my watermelons.

Kiva – Year 1 Report

Posted on by under Me

Pin-pointing the first time I was told that “the purpose of life is to pull people up rather than to hold them under” is a difficult one, but if I was pushed to pass on one piece of advice to another, that would be it. While this is, more or less, a restatement that we are standing on the shoulders of giants, with the implication that if you are one of the first to reach the new height then you (to risk of further straining the metaphor) help others to climb and to see in to the distance with you, it doesn’t necessarily make it redundant.

I appreciate that this is a fairly pretentious way to frame the sentiment that we should be good to each other, and never purposefully inhibit those that have done us no harm. However – and even the most ardent adherers to doctrine must agree – that “the game” at present is framed towards taking from others to further yourself, with the aim to amass greater and greater quantities of wealth, rather than collectively pushing forward with the interests of all.

Oh how simple it all sounds…

For me, charity has always seemed to be a partial solution to this problem (in fact it’s probably one of the better ways we’ll ever have), and exactly a year ago today, after a few months consideration, I decided to do something positive and created an account on Kiva.

Kiva is a micro-credit service facilitated by users lending money to entrepreneurs and students in 73 countries via a network of field partners. People sign up to Kiva, add funds via a Paypal account, and then use these funds to facilitate loans in multiples of $25 (up to a maximum of $500) to any available loan listed here. Once a loan is made, and fully funded, the money will be started being paid back – exactly like a typical loan – but while the field-partners charge interest on the loans that are made, neither Kiva or its users receive any additional payments (a point must be made here, if I were to put the money in to a flexible savings account, I would be earning a rate of interest probably not much better than 0%).

This means that for every $25 I deposit in to my Kiva account, the maximum I will ever receive back (bar extremely unlikely positive shifts in the exchange rate) is that ‘same’ $25, and much like any typical loan the lenders can miss payments or under pay (delinquency), or even over pay, or simply never pay the money back (default).

In the past 12 months these are my headline stats:

  • Total Loans = 58
  • Total Deposits = $829 (as to why this isn’t a multiple of $25 I’m not sure)
  • Total Lent = $1600
  • Total Repaid = $751.82
  • Number of Fully Repaid Loans = 10
  • Total Value of Fully Repaid Loans = $375
  • Total Outstanding = $797.18
  • Total Amount Lost = $1.00
  • Amount Lost due to Default = $0
  • Amount Lost due to Exchange Rate Changes = $1
  • Countries = 45 / 73
  • Total Refunded and Expired = $50 (this occurs when a loan is not fully funded within the time limit)
  • Number of Loans Delinquent = 5
  • Amount In Arrears = $8.96
  • Delinquency Rate = 1.12% (arrears / total outstanding)

So far this means that I’ve made loans to numerous people, with virtually all having a perfect repayment record. This spans from watermelon farmers in Ukraine, to a taxi driver in Azerbaijan, Child Care in Iraq and a fruit and veg stall in Timor-Leste. It needs to be mentioned that delinquency rates are actually quite volatile, and typically the lenders will correct these within 1 month, indeed only 1 of the 5 currently delinquent loans has been delinquent before, and this gives me a high confidence that these loans will correct themselves over the next 12 months and/or the end of the specific loan period. To give context, in January ’14 I am expecting repayment on 40 loans ($100.23), with 43 in February ($99.83) and another 43 in March ($97.29) with the vast majority paying back on time (if not slightly sooner). This ~$300 can be immediately relent and used to fund up to 12 new loans, and this experience over the last 12 months has given me the feeling that – while I am admittedly not making any money – I’m not losing any, and as I can withdraw the funds whenever I wish, or choose to re-lend them, I haven’t felt that I’m being “cheated” at all.

But when it comes to choosing loans, and as it is ultimately my money, I have two rules I stick by:

  1. A 50/50 overall gender-split on loans.
  2. All Field Partners must be secular in nature.

As it happens, neither of these two rules are hard to accomplish using the extensive options available as part of the built-in search functionality, and also those materials provided by the “Atheists, Agnostics, Skeptics, Freethinkers, Secular Humanists and the Non-Religious” Lending Team who’s goal is to promote secular values (by helping loans from secular Field Partners, especially in theocratic countries) and show that “we care about the suffering of human beings”. This team alone has so far made over $13.5 million of loans since August 2008, making them the single largest loaning team on Kiva – an impressive achievement in anyone’s books.

If you’ve made it this far, and are possibly interested in “having a play” with Kiva, and seeing just what you can do, it’s simply and all you need to do is click here…

http://www.kiva.org/invitedby/weal

and Sign Up – even better, if you do, both you and me will get $25 of free loan credit to go towards funding a loan.

Go on – do something different in 2014.

Mistakes

Posted on by under Business, Tech

We all try, in our own ways, to be perfect. But everyone makes mistakes.

The past couple of months have been an interesting time at Palringo. A few times a year the offices converge on Newcastle for a “reset meeting” where we discuss our plans for the next 3-6 months, re-evaluate our current commitments, and scrap anything we no longer believe in.

The “Invite Centre” was originally conceived by Martin Rosinski approx. 6 months ago and presented at one of these meetings. I was tasked with making the largest user-facing project of 2013 (so far) a reality. The “Invite Centre” is how we plan to grow the Palringo community – and is heavily related to the Reputation and Achievements system I have detailed before. The “Invite Centre” would be the “big ticket item” for Palringo iOS v6.4 and Palringo Android v5.10.

Palringo iOS v6.4 was released on Monday 2nd Sept, 2013, our previous major release, Palringo iOS v6.2, was released on Monday 22nd July, 2013 – with a follow up bug-fix release, v6.2.1, on Monday 29th July.

Some background; all major releases will have up to 2 big ticket items – these are tasks that require a lot of development and testing time, and may not be apparent to the user. For v6.2 this was a transition to CoreData. A change of this scale will always result in bugs; the simple mantra of “the bigger the change, the more bugs there will be” always holds true. You can never test every combination, or find every bug – this is part and parcel of software development, and shouldn’t be a surprise to anyone reading this – our game is to minimise issues, and begrudgingly accept that you’ll never eradicate every idiosyncrasy, and celebrate when you do finally manage to terminate that one little bastard that has been haunting you for months.

The need for v6.2.1 was apparent early on, both from the reports we were getting back from TestFlight – an iOS crash reporting system – and our in-app Support team. The scope for the bug-fix was defined, fixes found and made, and the update released 7 days later – an impressive feat, especially when taking in to account the Apple Review Process, and the time it takes to get an app reviewed.

For all parties, v6.2.1 was an improvement – but still had more issues than pre-6.2 releases. This is another issue with updating any core functionality for a piece of software. Once you get the big bugs out of the way, you spend years continually polishing the code, ensuring it is running as best as it can. This level of polish is not possible to replicate in a month, or two, when you have to change something significant.

The “Invite Centre” is getting closer, so we barrel on through, deciding not to make another bug-fix release.

Feedback from our Support Dept is that the users are living with the current issues that still arise in v6.2.1, which I incorrectly assume means they are happy. They are not.

This was my first mistake.

We continue, and as the week becomes two, and then three, the complaints from our Support Dept increase. It is already the 20th August, with v6.4 meant to be our “August Release”, there are minimal signs of submission soon.

A meeting is called and we – myself, the devs, and QA – detail the situation, explaining that we aren’t happy with the current state of v6.4 and the fixes we have included. We don’t believe the experience will be any better (“We’ve not so much fixed the bugs, as just moved them to other places.”) and that there are a number of known bugs. One of our external testers continually experiences this bug, but cannot find a way to reproduce, and neither can we when plugged in to the debugger. We diagnose this bug as being “severe but infrequent”, and decide that we can live with it. We agree, in part due to the ever increasingly vocal complaints from our users, that we would fix the high priority bugs, and submit the app in close to its current state, acknowledging it isn’t perfect, but will be better when make these fixes.

This was my second mistake.

Once we have made all the agreed fixes, we submit to Apple.

We spend the week between submission and release attempting to reproduce the “blank messages” bug (as it is now known), to no avail. The App is approved over the weekend – Apple didn’t find it either – and I wait for Support to confirm that nothing has happened over the weekend for them to insist on a delay.

While I’m waiting, we finally reproduce the bug. It only occurs under very specific conditions. Firstly, you had to have minimal memory available – which in Palringo happens if you are in many, many groups, receiving hundreds of messages a minute – and secondly you receive a message (group or PM) when not viewing the messages tab.

My choice is simple;

  • Pull the approved app, which we know is an improvement for most, and hope we can find a fix in a short enough window of time to please the community that are increasingly vocal in their complaints about v6.2.1, or,
  • We release, and bug fix if required.

It is my decision – and the fact we can now reproduce the bug does not matter. The submission passed our locks with us knowing the bug existed, without a caveat of “if we can reproduce, we’ll pull”. This is different to a bug you suddenly find before a release – that would cause a delay for at least as long as it took to triage. Fixing bugs is an on going aspect of software development, when you find out how to fix them is irrelevant, if you’ve decided that you are happy with said bug being present.

I release the app.

This was my third mistake.

It would ultimately transpire that 99% of our user base would not be affected by this bug, and that v6.4 is a noticeable improvement – but the 1% it does, it is a continually reoccurring nightmare.

We get reports of users experiencing the bug every 4 minutes, that the app is unusable, that they need to downgrade. We cannot roll back the app – Apple don’t let you – and we can’t re-submit v6.2.1 due to the approval delay. We add text to the “What’s New” section of the App Store Description, telling high-usage users to not upgrade to this version of the app, as well as passing out the message in-app.

We start frantically fixing the bug. Either there is a problem in the locks, or, we made a mistake.

“Release early, release often” is how we manage QA, nightly builds are increasingly common as we progress through the release cycle. Our bug-fix process barely differs from a standard cycle, and within a few hours we have a proposed fix and have pushed a beta release to our testers, with steps to re-create the situation. “Negative Testing” is some of the most time consuming software testing you can do – and is strongly related to the concept of proving a negative – and so all you can do is state that “these steps no longer seem to cause the problem(s)”. You can never really explicitly state the bug has be fixed.

I call a meeting Tuesday morning for Technical and QA, for 30 minutes we discuss just how we got here, why we are in the situation, have a full debrief, and discuss what we need to change. We have faith in our process, we admit, understand, and accept that the bug was misdiagnosed – we agree that, if we had to do it again, we would. We get back to work.

Our testers report back, things seem better…but different. We have new bugs. We apply yet more plasters, but it becomes clear that this is a fundamental issue with the framework.

The Technical Lead consults with me, and expresses his desire to rip out the back end, and completely re-implement the view. We decide that this is the best way forward.

A “functional but not pretty” beta is released with the old framework ripped out, and the stability improvements are apparent from the get go. A by product is that, because we removed the view completely, that specific bug can no longer exist.

Many more interim releases are made, and late Friday we make a final RC release to our beta team, and agree to submit to Apple in parallel. At the time of writing, we are awaiting approval.

—

So, what have I learnt?

  1. The balance between fixing bugs vs. new features is even more delicate that previously imagined; and it must be a balance, you cannot sacrifice one for the other.
  2. It is always better to delay something slightly, than make an action you cannot undo.
  3. Confirm your diagnosis of bugs is still true if presented with new evidence or data.
  4. If a bug is frequently encountered by one person, it will definitely be encountered by the many.
  5.  Everyone makes mistakes – but it is unforgivable to not learn from them.