Alignment

I’ve spent a lot of time thinking about alignment recently.  Over the years I have learnt that if we are not aligned, ideas fail no matter how good an idea is.

Testing

There was a time not that long ago where automated testing was not the norm.  I was fortunate enough to work for several years in the mid-2000s on a system covered with a fully automated test suite that deployed to production every 2 weeks or less.  It was, in retrospect, an incredible software development environment.

When I moved on from there, I joined a C# development company.  I wrote tests with my code.  I soon discovered that no one else in my team did.  They were mildly amused with the idea.  That was what manual testers were for.  Over time, as they were not invested in tests, if they touched code that was tested, the tests would break, but they would not fix them as they didn’t even run them so never knew they were broken.  We set up a CI server to compile and run the tests. Even then, the tests would still be broken by another developer and not fixed as the culture was not one that watched for red builds.

I learnt that if we are not aligned in our expectations we can waste a lot of time and still fail.  This makes adopting testing hard if everyone is not aligned.

It could take just one person not to care about red builds and broken tests to break everything – unless the team or the organisation corrects that behaviour.  In an aligned team that takes CI and testing seriously, hopefully that will not happen.

I’ve also seen a testing culture across multiple teams respond to new developers who did not test by showing them the way and giving strong feedback when they continued to avoid testing.  I have experienced that when the alignment and culture is well embedded, self-correction will occur.

Scrum

I worked in an organisation where I helped adopt Scrum to good effect.  If people are aligned to try and adopt and to openly discuss the issues, then we can be successful.  My experience in that case was that the most vocal sceptics became the most vocal advocates.

I moved at one point to another organisation which was kind of doing Scrum. No one was particularly worried about their process.  They did it. That was all.  I learnt if you are the only one who cares about trying to do something well, then there isn’t much point.  If no one else is aligned with doing something well, then as soon as your energy drops, no one else will pick up the slack. (I wrote about this in The Same Things Do Not Always Work.)

I learnt that if we are not aligned in our expectations I can waste a lot of my time and still fail.  This makes adopting Scrum and agile hard if everyone is not aligned.

It could take just one person to make Scrum very painful to adopt, particularly if that person is one with influence.  The team or the organisation needs to correct the behaviour to allow success.  In an aligned Scrum team that behaviour is harder to not confront, so hopefully one person is not able to do that.  In an un-aligned Scrum team, it can become a mess.

Agile Aligned

I have worked in an organisation that was highly aligned on agile software development ideas.  It was a pleasure to work there. The cultural expectation was “we do agile” and we argued about the nuances, not the should we do this thing at all.  This alignment was not just in the way we built software but also the way we interacted and were managed / autonomous.  This alignment made a huge difference to the quality of conversations around the problems we had to solve and how we worked.  Our core values and principles came from approximately the same place so the practices we advocated were based in those.

Alignment was clearly beneficial to the success and enjoyment of working together.

A Cycle

I’ve been through different parts of the cycle of discovery, adoption and mainstreaming of these ideas in different organisations.  I’ve been through where they have not worked, or they have not been personally worth it, and I’ve backed away and refocused.  And I’ve also been at the end where this is how we work and have seen the clear benefit of the alignment.

As we try new ways of working we keep on going through this cycle.  As we learn new product methodologies, new ways of managing / working with people, as we learn how much autonomy is a good thing for a given set of people, as we learn to experiment – and learn which things we need to learn and which things we know already (so why are we relearning them?), as we scale, we go through this cycle.

Sometimes an idea, principle or practice might be too soon for this group of people as no one else will help drive it, or sometimes it might be just the right time and we can make progress, or maybe it is the right time, but we are struggling as some might be disrupting.

Good pains or bad pains

Adopting new ideas often results in growing pains.  Aligning to working through those pains is important.  If not, we will likely fail.  If we are all aligned in our values and principles, maybe this alignment will be easier to achieve.

A common comment is: if Scrum shows up problems, do we abandon Scrum?   Or do we fix the problems that are shown up.

As we pick up new ideas, this pattern emerges again and again.  Is the idea the problem or are we failing to solve the problems that are being exposed?

The more, the harder

It is far easier for a single team to be aligned.  Alignment can always be helped through retrospectives.  This assumes that you’re aligned that retrospectives are useful and that everyone is working together to make them work.  Otherwise they might be potentially wasteful and negative.  Potentially one person could make a retrospective useless.

As the number of people grows that need to be aligned, alignment becomes harder.  As does the probability that there will be at least one person who might be disrupting – proving the exception to the alignment.

My current running hypothesis is that it can take just a few unaligned people to disrupt the adoption of any valuable idea – unless the team(s) or the organisation corrects the behaviour.  Strong alignment comes with a strong capability of team(s) to self-correct individuals who are trying to disrupt that alignment.

As the number of people involved grows how do we keep aligning across teams at the right level?  Could it be possible to structure things such that the number of people that need to align are fewer?  Would this make alignment simpler?  And what would it mean for the organisation at an even higher level?

Certainly it seems beneficial to build a robust, aligned culture that self corrects to stop the power of one from breaking it.

Alignment is hard.  People are hard.  But achieving alignment can be really worth it.

Advertisements

Getting data out of your system

We have data. It needs sharing.  What are the options?

One must assume that the data is required, so sharing it is necessary.

We could share the database: I’ve written about burning my fingers on doing this as well as the drawbacks of doing it if you’re forced to.  So what are other options?

If we do not provide SQL as our API, then we need to build something to provide that.

We could build an HTTP API to be called.  This API could provide the state of the data now, or we could attempt to provide a solution that provides a flow of events.

We could write data to another database and that is the contract that we share.  This still provides SQL as the API but at least the main application database can change at will as long as the contract into the secondary database is maintained and supported.

We could event the data and another service could listen to the events, reads the data and makes it available in another database and that is the contract with the consumer(s).

It is inevitable that something needs to write the data somewhere for the other application that wants it to gain access to it.  It will depend on what that application is as to what makes the most sense. But something will need to be built to provide the data.

There are many solutions that will hopefully keep your system changeable.  Find one that does not require large amounts of up front work, continuous maintenance, and provides the feedback needed to tells us if we are breaking our consumers.

Burning my fingers with database integration

On the whole integration databases lead to serious problems because the database becomes a point of coupling between the applications that access it. This is usually a deep coupling that significantly increases the risk involved in changing those applications and making it harder to evolve them. As a result most software architects that I respect take the view that integration databases should be avoided https://martinfowler.com/bliki/IntegrationDatabase.html

I have learnt this lesson painfully in the past.  The clearest example is an application which was exposing a new API to replace an older one in a large monolithic app.  The desire was to build a model that better represented the domain. The implementer was going to build it in a new rails application.  It would be the new API and it would represent a better view of the system. At the same time, it was viewed that this API would not change “too much” after it was released.  So when reviewing and conversing about the solution and how it would be maintained, the fateful “it is not going to change, so let’s move forward and not worry about maintainability now” was stated.

The decision in this instance conversed around sharing the model as a Gem between the two applications, or otherwise sharing it as views and having the new application consume the views.  Having a model as a Gem for a concrete implementation of data is not a great idea as it means that all consumers must always have the latest gem deployed as that is the latest representation of the database schema and hence any time a schema change happens all consumers containing the gem should be redeployed – even if the change has no bearing on a given consumer, they need to keep in sync so that they interact with the data correctly.  This is particularly true if the model includes code to interpret what is in the database columns meaningfully.

In this instance we chose to share views to share the data.  If we had multiple applications owning the schema we could get into trouble managing changes to the migrations from multiple sources.  To resolve this we chose to make the primary application own the database schema. That meant any view change required for the new API needed to happen in the primary application first.  It would also need to be deployed to live before the API code could consume it.

If you recall the “it’s not going to change” comment.  It turned out six months later that we thought this API was useful and started to use it – both directly and from a search service.  Long story short, we did change the data a lot and this design choice became a real pain.  Changes only needed in data for the API caused multiple repos to change and be deployed.  Deployment could be painful as sometimes the different database schemas were out of sync. Deploying an older version of the main application would downgrade the views for the API consuming it.  Testing could be painful as we needed a matching test database based on the main application so that the views were up to date for the tests to run in the test database schema.  There were lots of lots of little niggles due to the coupling of the implementation.

The worst part of this choice was when people moved into the project they struggled to understand the coupling.  They would forget the process as too many things were solved by “we should remember to do XX”.  The solutions were too often “humans be better”, not systems providing feedback when the humans inevitably did the wrong thing.  But it is hard to provide automated feedback about changing code in one repository that is affecting another one.

When we did this I was working with the best group of agile software developers I have worked with.  It caused us a lot of pain and we owned all parts of code accessing the database.  It is hard to get out of this path once you have started down it.

“As a result most software architects that I respect take the view that integration databases should be avoided” – Martin Fowler

In retrospect this really does continue to feel like the smarter choice.

Building a learning culture

Over the last couple of years, I’ve been focusing on how to push the skill / ability level of the teams that I’ve been working in.  For the last little while that has been focused across multiple teams.

Doing this from inside a team is “easy”.  Time can be spent pairing with anyone willing to pair with me.  To inculcate an attitude of understanding, of questioning, of challenging, and of trying new things to understand what it could look like.  In my experience, keeping an open mind and having respectful conversations enables much learning and growth in any developer – which in turn is more valuable for the company. And I keep on learning new things from developers of every experience level along the way.

Things are different when faced with the challenge of growing a learning culture across multiple teams and not being in any given team.  The following are some of my thoughts and experiments around building a learning culture in an organisation of 4 teams and growing.

Why promote a learning culture?

The software industry has a shortage of skills. It always seems hard to find good software developers to hire.

We have an ever-growing pool of new developers.  A quote going around is “Every 5 years the number of programmers in the world roughly doubles.  So half the programmers in the work have less than 5 years’ experience.” (1)

We are in an industry with a wealth of knowledge that can be consumed – but practical application is needed to truly understand the nuances.

We are in an industry where the learning gained from experience does matter.

Axiom: Experience is valuable.

However

“True learning involves a permanent change in the way you see and act in the world.  The accumulation of information isn’t learning.” – Benjamin Hardy (2)

True experience is more valuable

True experience involves a permanent change in the way you see and act in the world. Experience is not truly valuable unless it is learnt from.  True experience is most valuable when it can be understood in terms of principles and values that were effective (with a good experience) or were broken (for a bad experience).

Experience should be viewed around a common understanding of the values and principles being applied.  The values and principles should be based on the needs of the organisation.  Experience should be respected, discussed and challenged in line with these values and principles.

How can we harness experience to speed up learning?

Given that experience that we learn from is valuable.  How do we better harness the true experience in any given room / team / company?  How can we learn from our experiences and share those learnings with those who have not yet had them most effectively?  How can we extract the years of learning out of the experienced developers’ heads so that we don’t need to have 10+ years to learn it?

What about shared values and principles?

Hypothesis: Teams that value the same things in software will build software more effectively.

Lemma:
If we know vaguely* where you are going
We can all pull vaguely in the same direction

*Vaguely is important.

If we get too precise it limits a team’s ability to innovate and effectively solve the real problems they face.
If it is too ill defined, the teams have no direction and can waste effort duplicating work or going in different or unexpected directions.

Lemma:
Any decision we make should be based on a mental model that can be expressed.  If you can’t express the reasoning, then the reasoning is flawed.

If you don’t like something in a code review – understand why.  If you can’t express the value or principle that is being violated maybe you don’t know why and are just being opinionated.  Understand your opinion first, before expressing it.  It may be that the values and principles are still being met, just in a different way that you aren’t used to.

We need to move from conversations about how to conversations about why.  What are the intentional trade-offs and design decisions that are taking us in this direction or got us here? The how is important – but driven by a deep understanding of why.  This allows us to ensure there is no cargo culting of solutions and no sacred cows being ignored. How do we elevate the conversation from technology details to software truths?

Build a collaborative learning culture.

If we are self-aware and understand the decisions that we are making, then we can discuss these decisions with our teams and build a common understanding of what is a good decision for the team.  The team needs / context outweighs the individual’s needs.

If the team can understand the decisions it is making collectively, then teams can discuss their decisions with other teams and we can build a common understanding of what is a good decision for the organisation.  The organisation’s needs and context outweighs the teams’.

Context

It’s hard to justify why something is good or bad without context.

Without context – if the software does what was asked for, then it is good.  It doesn’t matter if it is spaghetti code.  It doesn’t matter if it is unmaintainable.  It doesn’t matter if it is inefficient.  Being right enough matters for now.  If the software never changes then it is good.  If the software changes, then we might have wanted to optimise for changeability and the current code is no longer good (enough).

Some things that we have tried

My focus has been on building a common understanding of why we do the things we do.  This provides a space for communicating opinions / values / ideas and increases understanding between the things that developers are valuing.

We tried

  • Code appreciation / Code review sessions
  • Code kata and conversation sessions
  • Kata sessions on Friday morning
  • Coaching – though harder to do cross-team
  • Retrospectives – though usually less focused on code

We introduced guilds to encourage cross-team conversations around specific topics.  These included: architecture, continuous delivery, security, databases.

We have an active tech blog, containing knowledge that is useful to remember or to share cross-team – e.g. security fixes made / to be aware of, continuous delivery pipelines of different teams, architectural knowledge base.  This is also a knowledge repository for learning so that new hires can start to get the context of what the rest of the team has already learnt and what their values are.

Code appreciation / code review sessions

This fluctuated through many different forms.  From once a week rotating through a different developer each week across all teams, to sharing just in your team and discussing and then a less frequent cross-team sharing session.

Code kata and conversations

1 hour a week facilitated session on different software topics ranging from TDD to DDD, from SOLID to testing practices and design patterns.  The focus is on practices and discussing and understanding the values and principles that elevate from the exercises.  This is usually in pairs or small groups working on a problem and then we retrospect on learnings as a group.  This started with doing some katas to explore certain ideas but then moved on to many different things and possibly shouldn’t have ‘kata’ in the title any more.

Learnings

We have been successfully changing the conversation from right and wrong and syntax to values and principles and design.  This has been noticeable across the teams.

When we agree on the values and principles, “right” and “wrong” become much easier to articulate.

I have relearnt that not everything is a teachable moment.  Different sessions have different focuses and sometimes in a group environment attempting to ask too many probing questions can be intimidating.

Not everyone will engage.  The key is to ensure that enough do and we focus on what the company needs from building a stronger learning culture and a strong software development team.  Hopefully the rest will pick up from the majority.

Where to from here?  Building a more collaborative learning culture

Everything that we’ve done so far has been to enable a collaborative learning culture.

But it has been focused on bringing the group involved up to a common level and I feel that we are now at a place with enough people engaged and interested.  We need to become truly collaborative in our learning.  This year I hope to see more presenters, more sharing and more growth and understanding across the teams.

The end result will hopefully be a competent, young team that can collaborate effectively around shared values and a common understanding about how to experiment and learn and discuss the right solutions for the organisation.

References:

(1) https://twitter.com/web_goddess/status/804452382536912897) – attributed to Robert Martin. Martin Cronje at Agile NZ 2016 quoted a similar figure.

(2) Via a great presentation by Katlyn Parvin – https://speakerdeck.com/katlyn333/am-i-senior-yet-grow-your-career-by-teaching-your-peers

(3) It looks like Martin is doing similar things to what we’re attempting to do (after moving to NZ) – https://speakerdeck.com/martincronje/agilenz-towards-mastery-establishing-craftsmanship-culture-in-a-team

Coding with confidence

Change is inevitable. It is the one constant in software development.

I find optimising to be confident in the face of this inevitable change valuable. I want to be confident that what I’ve done – what I’m releasing – really works.

When changing software I’ve come to ask several questions that help me to be confident in the changes that I make.

  • How will I know when something breaks? Does it matter if some given functionality breaks?
  • How easy is it to understand?
  • How localised would changing this be in the future in any new direction?
  • How confident am I? How can I be more confident?

This leads me to a list of principles that I currently value when writing software.

  • Feedback
  • Simplicity
  • Changeability
  • Ease of Understanding
  • Deliberate Intent

These principles lead me towards certain practices that I currently find useful.

Feedback
In order to know when something breaks a feedback mechanism is needed. Waiting for feedback from an end user that you messed up is far, far too late. The best time for that feedback is the instant the change was made. That is the time when the knowledge is known about what the breaking change is and hence how to fix it. This means testing. Testing at the right levels. Robust testing to ensure that the feedback is as clear, direct and immediate as possible.

Emergent design and Test Drive Development combine to provide feedback. The tests confirm that the implemented requirements are still working.

In order to build something that I’m unsure of, I want to do the smallest thing possible to validate whether what I’m trying to do will work. I cannot write tests when I’m unsure of how it will be called or of how it works. I don’t want to build something to discover that what I thought I was being called with isn’t actually want I’m being called with.

Unexpected things can happen. If I don’t expect them, I may not plan for them. But often I will implement ways to log or notify the unexpected – for when the unexpected happens. For instance using logs to log unexpected errors, or tools like Airbrake (for error catching and logging) or New Relic (for real time stats).

Simplicity
Systems become complex. Large balls of mud become very complex. If one assumes entropy in software is inevitable, any system will get messy. Instead of one large mess I try to focus on creating lots of small messes. Patterns I use to help towards small, contained messes include the Aggregate pattern; valuing Composition over Inheritance; separation of IO and CPU operations; Pipes and Filters; and Anti-Corruption layers.

Emergent design helps me keep a solution simple by only implemented the required functionality and allowing the design to emerge effectively behind the specific interface.

Changeability
I try to isolate the same domain concepts using DRY. But I try not to isolate accidentally similar concepts. As Sandy Metz puts it – prefer duplication over the wrong abstraction.

Test! Tests provide me feedback on the change that is being made. The expected tests should break. Tests that break unexpectedly should be looked at to understand the unexpected coupling in the system. I try to focus on increasing cohesion over reducing coupling. Increased cohesion provides a better-defined system – while also reducing coupling.

I aim to isolate change with design practices and test practices. The goal is to test at the right level. I try to test the interface so that everything below the interface can be allowed to emerge in whatever sensible design is needed for the requirements now. I encapsulate object creation in tests to allow the creation of the object to change over time without having to change a large number of tests. The goal is that any change in any direction is as painless as possible.  This doesn’t mean predicting the change, but rather isolating the obvious changes and hence limiting their impact when they do change.

There should be no area that is too scary or painful to change. When there is the goal becomes to find ways to reduce the fear or pain – assuming it still needs to change.

Ease of understanding
The ability of someone else (or yourself in 6 months time) to understand the existing code is highly underrated. Code is write once, read many. Every time an issue occurs near a given part of code, the code may need to be understood. If it is clear and obvious and is able to be trusted, it will be quick. If not, it could add 30 minutes or more to understand the code even if the code is not relevant to the problem. The cost of not being able to read and understand the code quickly is high over the lifespan of the system when considering the number of people who will read the code over time in their quest to make changes and solve problems.

I use Behaviour Drive Design to drive what tests to write. If I can’t write the test or the code cannot be built from the test, then I do not understand what I’m doing yet. The tests describe the intent of the code.

Discoverability – how will someone else discover that the code works like this? I try to ensure that breadcrumbs exist for a future developer to follow. If the knowledge exists for a developer to follow, then they can ask the question – do I need this.

I value explicit over implicit. If developers are joining your team, how will they know how the system works? Code is obvious. Implicit rules are not. Frameworks such as Rails come with a lot of implicit rules. These rules can be very opaque no matter how experienced the new developer is. You don’t know what you don’t know. The power of Rails is the weakness of Rails. But good Rails developers can move between Rails projects and know the implicit rules. But what if there are implicit rules that are specific to the code base? Those are hard to know. Again – you don’t know what you don’t know… until you realise that you didn’t know it and someone tells you – or you go really deep. Either way a lot of time can be unnecessarily lost grappling with the unknown.

Deliberate intent
I try to make conscious, well-reasoned design decisions. My goal is to keep the exposed interface tight and deliberately defined. For any solution I try to think of at least three ways to solve the problem and then make a choice based on the pros and cons. This ensures that I’m thinking deeply about the problem instead of simply implementing the first design thought that arose in my mind.

A continued experiment
These are my principles and practices that I’m experimenting with. Over time I imagine they may change or modify as I continue to experiment and learn.

In order to experiment I generally try applying a constraint and see what it does to the code base. Do we like it? Does it make things better? How do we feel about the result? To paraphrase Kent Beck – If it doesn’t help, stop doing it. If it does, do more!

I continue to look for the experiments that I can do to make things better along with the underlying principles and values that they represent. I hope to blog about some of those experiments in the future.

There is a lot more that could be said about each of these principles. I hope to follow up with related posts digging into different implementations and designs that I have derived from these principles.

 

Credit
Credit goes to Kent Beck’s Extreme Programing Explained that focuses on Values, Principles and Practices. I find this an incredibly useful way in which to view software development.

Personal Responsibility

Last week I was able to attend Agile Africa 2015. The most inspiring talk, amongst the many great talks and conversations, was Kent Beck’s closing keynote. He talked about the engineering culture at Facebook and its key overriding value of taking personal responsibility.

At Facebook as Kent Beck described it
The task you pick up is your responsibility to get right. All parts of it. Even when it is the first task you do when joining. There is no one else other than the developer who is responsible for the decisions required to get it done. If you deploy a bug, someone else may see it and will fix it. And they will give you feedback. Everyone is responsible.

When joining Facebook, developers join Facebook Engineering. Developers are not hired into a specific team at Facebook. In the first weeks the new hire chooses which team they will join. They are personally responsible for the choice to ensure that they join the right team and make the most impact in the organisation.

This is a culture that trusts you to know what you should be doing. If you get it wrong, you will receive feedback.

A culture
A culture of high focus on impact guides developers to prioritise their time effectively.

A culture of clear, direct and immediate feedback makes sure that there is a feedback loop on responsibility.

A culture of P50 goals drives improvement and striving for new thinking. P50 goals are goals that you aim to hit with 50% probability. Spread them around so that all the 50% probability isn’t in one area – otherwise no goals may be achieved at all! [1]

This is a culture of high responsibility, but associated with that must be very high trust. And it has scaled to thousands of developers.

Not Agile
Facebook does not claim to be Agile. There are pockets where there are lots of tests. There are even larger pockets where there are none.

What? The inventor of XP works at a non-Agile organisation? What does that mean? Is Agile dead?

Very agile
However Kent Beck has seen FB turn on a dime. FB is highly successful. Engineers enjoy working there – at least Kent seems to enjoy it immensely. It sounds like a very agile organisation.

Personal Responsibility
The key is leveraging personal responsibility. It is what Kent says he was getting at when he wrote the XP book.

High personal responsibility leads to developers needing coping mechanisms so that they don’t mess up. If you mess up, you own the mess – 100%. If you mess up too much, you won’t last long. If someone sees your mess, they will fix it – because they too are responsible – and you will get clear, direct and immediate feedback to act on for the future.

Personal Responsibility drives behaviours. It may lead you to TDD. It may lead you to lots of acceptance testing. It may lead you to implementing lots of metrics or monitoring a lot and responding as soon as errors are detected. All of these help a developer keep a handle on their work and hence help them to be responsible. There are no ‘the system is the fault’ types of answers to failure.

Personal responsibility leads to developers having to take control of the code base they are working on. It leads to developers being more confident of their work. If they are not, they fail. They will receive clear, direct and immediate feedback. And they must take responsibility for the problem.

XP has been pointing at personal responsibility. It also seems that personal responsibility can point straight back at XP as one of the possible outcomes of dialling personal responsibility to the extreme.

Personally responsible
Personal responsibility touches at the heart of much of great software development. It feels like it should be a core value to optimise around. I certainly am going to be thinking harder on what it means in everything I do.

If you weren’t there – watch out for the videos when they are out. It was a really inspiring (and funny) talk.

If you have seen the talk, let me know what you’re thinking as it would be interesting to hear what others will do around this.

 

[1] More on P50 goals in Growing Agile’s latest newsletter at http://eepurl.com/bxmJFb.