Building a learning culture

Over the last couple of years, I’ve been focusing on how to push the skill / ability level of the teams that I’ve been working in.  For the last little while that has been focused across multiple teams.

Doing this from inside a team is “easy”.  Time can be spent pairing with anyone willing to pair with me.  To inculcate an attitude of understanding, of questioning, of challenging, and of trying new things to understand what it could look like.  In my experience, keeping an open mind and having respectful conversations enables much learning and growth in any developer – which in turn is more valuable for the company. And I keep on learning new things from developers of every experience level along the way.

Things are different when faced with the challenge of growing a learning culture across multiple teams and not being in any given team.  The following are some of my thoughts and experiments around building a learning culture in an organisation of 4 teams and growing.

Why promote a learning culture?

The software industry has a shortage of skills. It always seems hard to find good software developers to hire.

We have an ever-growing pool of new developers.  A quote going around is “Every 5 years the number of programmers in the world roughly doubles.  So half the programmers in the work have less than 5 years’ experience.” (1)

We are in an industry with a wealth of knowledge that can be consumed – but practical application is needed to truly understand the nuances.

We are in an industry where the learning gained from experience does matter.

Axiom: Experience is valuable.

However

“True learning involves a permanent change in the way you see and act in the world.  The accumulation of information isn’t learning.” – Benjamin Hardy (2)

True experience is more valuable

True experience involves a permanent change in the way you see and act in the world. Experience is not truly valuable unless it is learnt from.  True experience is most valuable when it can be understood in terms of principles and values that were effective (with a good experience) or were broken (for a bad experience).

Experience should be viewed around a common understanding of the values and principles being applied.  The values and principles should be based on the needs of the organisation.  Experience should be respected, discussed and challenged in line with these values and principles.

How can we harness experience to speed up learning?

Given that experience that we learn from is valuable.  How do we better harness the true experience in any given room / team / company?  How can we learn from our experiences and share those learnings with those who have not yet had them most effectively?  How can we extract the years of learning out of the experienced developers’ heads so that we don’t need to have 10+ years to learn it?

What about shared values and principles?

Hypothesis: Teams that value the same things in software will build software more effectively.

Lemma:
If we know vaguely* where you are going
We can all pull vaguely in the same direction

*Vaguely is important.

If we get too precise it limits a team’s ability to innovate and effectively solve the real problems they face.
If it is too ill defined, the teams have no direction and can waste effort duplicating work or going in different or unexpected directions.

Lemma:
Any decision we make should be based on a mental model that can be expressed.  If you can’t express the reasoning, then the reasoning is flawed.

If you don’t like something in a code review – understand why.  If you can’t express the value or principle that is being violated maybe you don’t know why and are just being opinionated.  Understand your opinion first, before expressing it.  It may be that the values and principles are still being met, just in a different way that you aren’t used to.

We need to move from conversations about how to conversations about why.  What are the intentional trade-offs and design decisions that are taking us in this direction or got us here? The how is important – but driven by a deep understanding of why.  This allows us to ensure there is no cargo culting of solutions and no sacred cows being ignored. How do we elevate the conversation from technology details to software truths?

Build a collaborative learning culture.

If we are self-aware and understand the decisions that we are making, then we can discuss these decisions with our teams and build a common understanding of what is a good decision for the team.  The team needs / context outweighs the individual’s needs.

If the team can understand the decisions it is making collectively, then teams can discuss their decisions with other teams and we can build a common understanding of what is a good decision for the organisation.  The organisation’s needs and context outweighs the teams’.

Context

It’s hard to justify why something is good or bad without context.

Without context – if the software does what was asked for, then it is good.  It doesn’t matter if it is spaghetti code.  It doesn’t matter if it is unmaintainable.  It doesn’t matter if it is inefficient.  Being right enough matters for now.  If the software never changes then it is good.  If the software changes, then we might have wanted to optimise for changeability and the current code is no longer good (enough).

Some things that we have tried

My focus has been on building a common understanding of why we do the things we do.  This provides a space for communicating opinions / values / ideas and increases understanding between the things that developers are valuing.

We tried

  • Code appreciation / Code review sessions
  • Code kata and conversation sessions
  • Kata sessions on Friday morning
  • Coaching – though harder to do cross-team
  • Retrospectives – though usually less focused on code

We introduced guilds to encourage cross-team conversations around specific topics.  These included: architecture, continuous delivery, security, databases.

We have an active tech blog, containing knowledge that is useful to remember or to share cross-team – e.g. security fixes made / to be aware of, continuous delivery pipelines of different teams, architectural knowledge base.  This is also a knowledge repository for learning so that new hires can start to get the context of what the rest of the team has already learnt and what their values are.

Code appreciation / code review sessions

This fluctuated through many different forms.  From once a week rotating through a different developer each week across all teams, to sharing just in your team and discussing and then a less frequent cross-team sharing session.

Code kata and conversations

1 hour a week facilitated session on different software topics ranging from TDD to DDD, from SOLID to testing practices and design patterns.  The focus is on practices and discussing and understanding the values and principles that elevate from the exercises.  This is usually in pairs or small groups working on a problem and then we retrospect on learnings as a group.  This started with doing some katas to explore certain ideas but then moved on to many different things and possibly shouldn’t have ‘kata’ in the title any more.

Learnings

We have been successfully changing the conversation from right and wrong and syntax to values and principles and design.  This has been noticeable across the teams.

When we agree on the values and principles, “right” and “wrong” become much easier to articulate.

I have relearnt that not everything is a teachable moment.  Different sessions have different focuses and sometimes in a group environment attempting to ask too many probing questions can be intimidating.

Not everyone will engage.  The key is to ensure that enough do and we focus on what the company needs from building a stronger learning culture and a strong software development team.  Hopefully the rest will pick up from the majority.

Where to from here?  Building a more collaborative learning culture

Everything that we’ve done so far has been to enable a collaborative learning culture.

But it has been focused on bringing the group involved up to a common level and I feel that we are now at a place with enough people engaged and interested.  We need to become truly collaborative in our learning.  This year I hope to see more presenters, more sharing and more growth and understanding across the teams.

The end result will hopefully be a competent, young team that can collaborate effectively around shared values and a common understanding about how to experiment and learn and discuss the right solutions for the organisation.

References:

(1) https://twitter.com/web_goddess/status/804452382536912897) – attributed to Robert Martin. Martin Cronje at Agile NZ 2016 quoted a similar figure.

(2) Via a great presentation by Katlyn Parvin – https://speakerdeck.com/katlyn333/am-i-senior-yet-grow-your-career-by-teaching-your-peers

(3) It looks like Martin is doing similar things to what we’re attempting to do (after moving to NZ) – https://speakerdeck.com/martincronje/agilenz-towards-mastery-establishing-craftsmanship-culture-in-a-team

Know where you’re going

A core idea in agile software development is that we don’t know enough right now so we should only build enough for what we actually do know right now.  This is sensible.  This idea underpins evolutionary / emergent / test driven design – allowing them to thrive.  If we can safely and confidently change the system in any direction from where it is now, then we are in a good place.  We are keeping the XP cost of change curve as flat as possible.

But the cost of change may become high if we need to continuously rework everything. If the knowledge is available to help make more informed design choices earlier, we should use it.

Know where the code is going

Good software developers are intentionally trying to make the code base better.  This is highly desirable.  Good software developers will refactor confidently and allow new designs to emerge.  When you have 6 good developers refactoring the same code base into what in their minds is the best solution it is plausible that you may land up with 6 very different designs going in 6 very different directions.  This might not be ideal.

What would make those 6 good developers into a great team is a shared understanding of how the code is being refactored so that we don’t land up with 6 different designs, but rather one collaborative design that everyone has contributed to – pulling it in the same general direction.

As a team, discussing about where we’re refactoring the code to is important.  A given pair might not reach the best design while working in this sprint and another pair may augment and grow that design in the next sprint.  If we share where we’re going, we can hopefully get somewhere in the region of the shared destination.  And we can more meaningfully converse about changes in design as the requirements change and agree on the next destination.

But don’t inhibit innovation and new ideas

The counter to this is that if a good developer is building something and they choose to build something different to the current design – maybe it is simpler, better, cleaner – this should not be inhibited.  Do not strive for uniformity and accidentally crush the very innovation that you need to channel from a good developer in order to have a great team.

Know where the business is going

A common way of breaking up stories is around the CRUD screens / actions.  It is sometimes easy for a Product Owner to define a series of screens that allow them to administrate some set of information. It is plausible that these screens are built without a conversation of why the screens are being built.  Possibly this is because the PO doesn’t yet know the exact details on how these things are going to interact.  But if this is the case, it is plausible that when the PO finally specifies how these CRUD things are going to interact in a meaningful way that the team could turn around and suggest that, based on how those screens have been built, that isn’t possible – or at least it is very hacky and we’re already building a legacy system with bad design.

Knowing something about where the business is going in the next sprint or 3 is useful.  Use it to track how the design is evolving and hopefully take this at least somewhat into account when designing the code.

But don’t future proof too soon

Don’t worry too much about planning for future needs that are trivial to add in a future sprint.  We don’t need to future proof the design.  We only need to know where it might go so that we can perhaps go there more easily later.  We need to know that the short/medium term goals will be supportable by the design we are creating today.  If it isn’t, it doesn’t matter too much, but the cost of change will probably go up if we need to massively refactor the entire system every sprint – so what can we do to help avoid that?

Know where the architecture is going

Systems are continuously evolving.  As architectural issues arise, an idea of what should be done to solve them should be discussed.  If we can agree on the direction that we need to move the overall architecture and understand what it could look like, then any team may be able to start building that within existing work that they have, in an opportunistic way.  If we have no clue where we would like to go then any team will probably continue to do exactly what they are doing now.

Alternatively, any team may start to innovate in different directions and again we may land up with the architecture of the system being pulling in several different disparate directions which may not be a beneficial in the long run.

If your teams discuss and plan what the architectural changes could look like, then we know where we’re going and we can start working out the baby steps to get there.  Those baby steps may be started to be done inside current work.  But without any clue of where we going we can’t even start to take any baby steps.

But don’t be too constraining

Architecturally knowing where you’re going is important – but the micro details shouldn’t be constrained.  It shouldn’t matter if we use redis or memcache in terms of an architectural solution.  Let the teams converse and decide as a group on the actual solution details when they are needing the solution and allow that to be based on the real experience and constraints they are being experienced in their teams.

Know sort of where you’re going

The precise destination isn’t important – the general one is. The details will vary.  The overview of the destination over the next 2-3 months hopefully will grow and evolve.  Despite that, hopefully it will still be comparable enough to call reasonably the same.

How abstract the destination is depends on the need.  For instance, it might be useful to sketch out the boundaries of several micro-services to have some idea of the destination and to give teams an idea of what they could break out of a larger system.  But don’t be married to those decisions if they turn out to be incorrect.

Use the knowledge just in time

Knowing the destination today should inform the code as a potential destination to refactor towards.  We shouldn’t be focusing solely on the road directly in front of our feet.  We should be looking up into the horizon and gleaning any knowledge we can of what is up ahead.

Knowing where we are going will allow the whole team to pull the code and system in a similar direction.

Knowing sort of where we are going should lead to an informed just in time decision about when to do something with that knowledge.

DDD, Aggregates and designing for change

I have been introducing several domain driven design patterns to the teams that I work with over the last while.  In doing so I have been struck by the repetition of a core principle for many of the DDD patterns.

An example – the aggregate pattern
The aggregate pattern focuses on the business domain, attempting to answer the desire for a software developer to work with fully formed domain objects that interact with each other in a meaningful way.

The aggregate pattern defines the aggregate as a combination of objects that interact as a single unit.  The aggregate is only interacted with via the root of the aggregate.  This ensures that the aggregate root can maintain the integrity of the whole aggregate.

If all interactions with the aggregate are via the root aggregate this means that we can control the implementation of the aggregate behind the interface of the root aggregate.  The aggregate should be designed as a cohesive unit and is decoupled from the rest of the system via the aggregate root’s interface.  The aggregate root’s interface is the API to the aggregate.

Tensions
The complexity is in keeping the aggregate small while useful.  Tension exists when the aggregate gets bigger. The aggregate interface can become complex due to the constraint that all access must occur through the root aggregate.  This tension may push one towards designing smaller interoperable aggregates rather than allowing a large ball of mud to be formed.

Unidirectional?
Just as a design choice could be to have unidirectional models; it may also be an interesting design choice to build an aggregate with unidirectional models – with the root aggregate being the model that knows about (potentially) all the child models.  This may be useful.

My personal preference is to try to keep things unidirectional for as long as it is sensible, but as soon as it isn’t sensible allow connections in both directions.  The aggregate root is controlling the access to the objects in the aggregate.  Allowing objects in the aggregate to know about each other bi-directionally increases the complexity of the aggregate as a whole but, assuming a reasonably small aggregate, the principle of small contained messes is still supported behind the interface exposed by the aggregate root.

The core
Circling back to the core principle for many of the DDD patterns – the factory pattern, the repository pattern and the aggregate pattern all define mechanisms for creating a well-defined interface and hiding the implementation details behind the interface.

The factory pattern provides a creational interface to build an object.  We get well formed objects from it and don’t need to know how they were formed.

The repository pattern provides an interface that abstracts away object storage / query.  We ask it a question and get objects back.  We tell it to persist something, and it happens.  We do not need to worry how.

The root aggregate in the aggregate pattern provides an interface that abstracts away the implementation of the aggregate.

These patterns make code simpler and reduce complexity by clearly defining what should go behind the interface and what should not.  These patterns allow the caller to not worry about the implementation details behind the interface.  All of these patterns support the idea of smaller messes.  If the implementation behind the interface is a little messy, it doesn’t matter, as long as it can be refactored safely later and the caller is not influenced at all.

Decouple the interface that the outside world uses from the implementation underneath.  And know why the code is placed behind the interface.  Design a cohesive unit behind the interface.  This is also how TDD encourages code to be written, assuming you can design the interface well.

In practice
When discussing domain driven design versus other design ideas and using test driven design to build software, there are often questions around which pattern to use or how they should interact.  These questions are often attempting to get black and white answers to a complex contextual problem that probably has many shades of grey in it.

What has struck me most about introducing DDD after introducing TDD and emergent / evolutionary design is that the core principle is to contain the messes behind the interfaces.  Have good interfaces and decouple them.  And ensure the implementation behind the interface is cohesive and can change freely as needed.  That is the core of software design and most patterns.  How do I change later?  How do I keep in control of the code so that when change comes (and it will come! Especially in unexpected ways) it is not accidentally impactful on the rest of the system.

Focus on embracing change
Worry less about the patterns, than what the patterns are trying to teach you.  Use the patterns, understand them, they are a language that can be used effectively among developers.  But the patterns are not the end game, the changeable system that they encourage is.

Keeping it Simple – unidirectional models

Keeping code simple is an art.  It may require a reasonable amount of work – refactoring and trying design experiments to keep it as simple as possible.  Changes to requirements may result in the current design no longer being the simplest thing any more.

One way to attempt to keep things simple is to reduce complexity by not implementing anything that a current use case does not need.  That way, changing the design only has to worry about code that is being used – there is no guessing game.

Another way to attempt to keep things simple is to reduce the paths that can be taken through your code base to reach a given point.  Making small cohesive units that can be reasoned about with a well-defined interface to the rest of the code base ensures you can change those cohesive units more easily.

Adding complexity

A way to increase complexity is adding unnecessary code.  This is often done because “it’s obvious that it is the right thing to do” or “we’ll probably need it” rather than driven from any actual use case.

An example of this can often be seen when using an ORM.  Given a model that is composed of attributes and a list of another type of model, a common pattern is for both sides to know about each other.  The parent model will have a list of children and the child model has a reference back to the parent model.  This is particularly obvious when the design is data driven (driven from the database tables) and the designer feels comfortable just grabbing any table / model and loading something from it and working from that view of the data.

A classic modelling example is an implementation of an Order.  An Order may have many Line Items in a list.  In the standard way that typical ORMs encourage you, the Order model will know about its Line Items and a Line Item will know about its Order.  This means that I can load an Order object and expect to be able to access its Line Items.  It also means that I can load an individual Line Item and directly access its Order.  And then I could use that Order to get the Line Items, one of which was the original object that started the request.

What’s wrong with that?

In the simplest implementation with limited complexity this may be fine.  But it could lead to some problems as the complexity of the solution grows.

Cyclic dependencies

One problem that can arise is cyclic dependencies – where the parent can call a child and the child can then call the parent and around we go.  This may be hard to reason about – particularly as the object graph grows in size.

Maintaining a cohesive world

These convenience methods increase complexity.  Having a design that can be entered from anywhere and needs to remain cohesive in any direction from that entry point increases complexity.  Additional code may be written to compensate.

It can lead to needing the world to be cohesive no matter which object I load and manipulate.  By allowing the Order to know about the Line Item and the Line Item to know about the Order, allows the option to add a Line Item separate from an Order, which may not be valid in the domain.  We may always expect a Line Item to have an Order and this could be violated.  In order to resolve this, we need to add validation on the Line Item to ensure there is an Order.

A potential design may be that changes to a Line Item may expect changes on the Order model.  If we can load the Line Item first, then we need to solve the consistency of the Order model.  Clearly we should add callbacks when the Line Item model saves in order to ensure the Order is up to date.

Maybe we have a tax amount stored on the Order.  When this updates, all Line Item’s need their totals to be updated due to the change in the tax percentage.  Clearly we should add callbacks when the Order saves to update this.

If I can access a Line Item directly, I can also change its total.  This may require the Order’s total to be updated.  Clearly we should add callbacks to update the Order when the Line Item is saved.

Now I’ve written a bunch of callbacks in order to keep my domain valid because I can load anything and use it.  I also may have introduced a cyclic dependency and a cascade of DB updates that I don’t control very well.  When I save the Order, all Line Items will be loaded and saved.  Each Line Item may change the total value in the Order… and if we do this badly we’ll get stuck in an unexpected loop.

Increased testing for unsupported use cases

In an ideal world, every use case that you build should have a test.  Putting in the back connections between two models should be driven by that use case.  If you don’t have a failing test case, don’t write the code.  If you don’t have a use for the code… don’t write the code.  But if you’re writing the code, write the test.  Which increases the testing burden – for unsupported use cases.

Containing complexity

Instead of providing inconvenient convenience methods, why not contain the complexity and not provide it?

Unidirectional linkages

For a start, could we make all models talk in only one direction?  What would that do to the code?  Suddenly we have reduced complexity.  In order to load one model, it can only be accessed one way.  There is no expectation of the other way.

For instance, assume that the Order knows about its Line Items.  But the Line Item does not have a back reference to the Order.  What would that mean?

A Line Item can never be created without an Order.

A Line Item is updated through the Order, therefore any consistency that the Order needs to maintain due to changes in the Line Item are simply done in the Order.

When the Order’s tax is updated, we now have a design decision – do we update all Line Items at Order save time?  Or do we update Line Items as we pass them out of the Order?  Suddenly we have control over the performance characteristics of this update.

The Order / Line Item relationship is now easier to maintain and reason about as the expectations of the relationship are simpler.

Line Items may still be read directly, but the design states that there is no expectation of accessing the Order.  In order to access the Order, use the order_id that may be in the Line Item object to directly load the Order and get a fresh object.

But that is more work!

Suddenly we have a little more work.  The little more work is clear and obvious.  We can’t load a Line Item and expect it to just get the Order.

I would argue that the backlinks have similar (and potentially significantly more) complexity – but that isn’t visible until later when you realise that the Order needs updating when you save the Line Item that you loaded directly.

Not putting in the back links acknowledges the real work to be done and the thinking required instead of pretending it isn’t there.

Let’s go further – the Aggregate Pattern

What would happen if we defined a model like the Order / Line Item relationship, but only allow any interaction to the Order / Line Item cohesive unit via the Order?  What would happen if when we interact with the model, it was always fully loaded and well defined and hence our expectations would be well defined.

This is the aggregate pattern, a pattern popularised by domain driven design.  The Order model would be the root of the aggregate. A Line Item would not be accessible to the domain except via the Order.  A design choice could be to only allow new Line Items being created via the Order.  Suddenly we have control of all interactions with the models.

There is more to the aggregate pattern and there are more implications and ideas, but I’ll write on that another time.

Keep it simple

Why not try to keep things simple instead of assuming things are necessary?  Experiment with patterns that have “extreme” ideas, as maybe there is something to learn.  Choosing constraints in your design that help simplicity and reasoning – even when it means slightly more explicit coding – can be a liberating thing.

Maintain your confidence in the face of change.  Keep it simple.

 

A potential caveat

Maybe your ORM needs the backlinks to do some of its magic.  If that is the case, it feels like it might be unexpected.

A design discussion, some options, and unidirectional models

When writing about discoverability in my last two posts, I came across the below code which violated that principle.  It was doing some meta programming in order to attempt to avoid repetition and to be more DRY.

def base_value_for(key)
    case info.send("#{key}_base")
    when 'value1'
        value1
    when 'value2'
        value2
    else
        raise ArgumentError, "Base keys must be value1 or value2"
    end
end

The caller then would be able to find the 3 base values by passing them into the base_value_for method.  The methods one_base, two_base, three_base are known and exist on the info object which is receiving the public send.  The UI has ensured they are set to either ‘value1’ or ‘value2’.  The current object has a method to return the value for value1 and the value for value2.  So this code is essentially allowing the initial base value for a calculation to be configured.

The caller would then call one of base_value_for(‘one’) OR base_value_for(‘two’) OR base_value_for(‘three’).  Calling with any other value that didn’t result in a method on the info object for key_<value> would blow up.

What is wrong with this?  Take a read here.  TL;DR – if you’re in the info class and search for ‘one_base’ in the code base there will be no search hit that tells you that this code is being called from here.  So you may assume it isn’t being called and refactor it.  The principle of discoverability is violated.

This code was written to avoid writing:

def base_for_one_base
    case info.one_base
    when 'value1'
        value1
    when 'value2'
        value2
    else
        raise ArgumentError, "Base keys must be value1 or value2"
    end
end

and repeat for two and three.

An alternative to the above non-DRY implementation is

def base_for_one_base
    base_for(info.one_base)
end

private
def base_for(info_base)
    case info_base
    when 'value1'
        value1
    when 'value2'
        value2
    else
        raise ArgumentError, "Base keys must be value1 or value2"
    end
end

Now the difference between base_for_one_base and the version for two and three is just a single line.

def base_for_two_base
    base_for(info.two_base)
end

def base_for_three_base
    base_for(info.three_base)
end

This is probably DRY enough.  We could probably stop here.

An alternate design

The existence of the info model is to hold the knowledge about the configuration.  So an alternative design could be to push the decision logic into the info model.

The problem with doing this is that the method is also using the value1 and value2 methods on the parent model.

The one solution is to Just Use It.

# info class
belongs_to: parent

def base_for_one_base
    base_for(one_base)
end

private

def base_for(info_base)
    case info_base
    when value1
        parent.value1
    when 'value2'
        parent.value2
    else
        raise ArgumentError, "Base keys must be value1 or value2"
    end
end

This sets up a cyclic dependency.  The Parent model has an info model.  The Info belongs to the parent model.  The info can be loaded and accessed by its parent directly.  The parent can be loaded and accessed by the info model directly.  But what if code were written that caused code to be accidentally called recursively between the two models?

On this small scale, it might be okay.  On a larger scale it might be much harder to reason about across several models.  And multiple models may be doing this in a highly interconnected way – and you then may have one large ball of interconnected models – or mud.

Unidirectional models as a design constraint

What would the code look like if we didn’t accept cyclic dependencies?  If all interactions with a model were always from a parent to a child.  That could make the code simpler in the long run as at least half the number of linkages would exist.  One for each direction vs. one for the one direction.

The code in this example would then look like:

# parent class
def base_for_one_base
    info.base_for_one_base(value1, value2)
end

# info class
def base_for_one_base(value1, value2)
    base_for(one_base, value1, value2)
end

private

def base_for(info_base, value1, value2)
    case info_base
    when 'value1'
        value1
    when 'value2'
        value2
    else
        raise ArgumentError, "Base keys must be value1 or value2"
    end
end

The negative here is we need to pass in the data from the parent that the child needs.  Perhaps that data shouldn’t live on the parent, but rather move to the child.  That might be a better design and our code is telling us that, but it was something we didn’t want to move yet.  However that knowledge is not exposed to the consumer.  The implementation of getting the rate for the consumer is only via the parent.

Simplicity

I favour one-way connections over bi-directional ones as they are simpler to reason about.  They result in cleaner, contained messes that are easier to understand (and potentially replace).

This contrasts against the effort required to break down the keen desire to have bi-directional connections that Rails encourages and makes so easy.  SQL data integrity also desires bidirectional connections.

It might result in jumping through some hoops to achieve.  But then the design is speaking and the constraint is possibly telling me that the design of the data might need to change.

I favour smaller, contained messes to make reasoning and understanding of the code easier.  One way to achieve that is through unidirectional models.

Discoverability – a design choice

I recently blogged on discoverability being a naming choice. I talked about how the choice of name may make changing it later easier or harder.  What would happen if we start using the qualities of dynamic ruby to do some meta programming – how would that influence a future developer’s capability to discover how the code works?  How does this break the model of “Find in Files” discussed in that post.

Let’s start by making it worse

Imagine an Active Record model, Tour, with a column full_price_for_tour in the DB.  In a vanilla rails project, searching for full_price_for_tour in the codebase may result in no hits at all.  Equivalently, with looking at a caller that calls full_price_for_tour on an instance of Tour, we will not find any reference to full_price_for_tour in the class file for Tour.  For new Rails developers this can be very confusing.

The programming model that the active record implementation is helping us with is potentially a useful one – dynamically determining the methods from the database and creating them on the object.  But it is harming discovery of how the code works.

So how do we help developers discover where the code is?

In a Rails codebase the annotate gem comes to the rescue.  It annotates the model classes based on what is in the DB for the matching table.  This allows a developer to discover the list of dynamically created methods that they can call on the model object – and hence what data the model object does expose.  This is a Good Thing.

Searching for full_price_for_tour will have a match in the Tour class file – as a comment in the annotations.  The developer now knows this method is the column in the DB as the annotations are allowing that discovery.

And then someone gets clever

The Active Record implementation leverages the dynamic qualities of Ruby to do something useful for developers.  All dynamic meta programming may not always be beneficial.  There are always trade-offs in software design.

Some production ruby code I saw recently implemented something along the lines of:

  def self.method_missing(method_sym, *arguments, &block)
    @hash_obj[method_sym]
  end

This code was written to provide helpers to access a hash’s properties with method calls.  This was a convenience method.  And there were some other helper methods defined in this class in order to work out some standard answers to questions that the hash provided.  On the face of it, this looks like a clever use of ruby as a dynamic language.

But how does another developer discover all the methods that this class responds to?  We need to find the hash definition to discover that.  In this case, the hash was from some json from a HTTP POST.  The simple question of what can we expect this object to answer to was not codified in the code at all leaving all developers on the team very unsure what the correct method names were.

Going back to the refactoring example.  When, assuming this was the Tour class and we were calling full_price_for_tour on this object – how would we find out what the implementation was?  First we’d fail to discover one with a “Find in files” type search.  Then we would have start spending some time working out why there wasn’t one and what the magic was that made it work.  As a developer this is time wasted.  Even worse, when the question is “what is the full interface”.

Another clever thing

Some ruby code I can easily imagine is:

  def self.method_missing(method_sym, *arguments, &block)
    method_as_string = method_sym.to_s
    split_methods = method_as_string.split(‘_’)
    obj_to_call = split_methods.shift
    method_to_call = split_methods.join(‘_’)
    obj = self.public_send(object_to_call)
    obj.public_send(method_to_call)
  end

Assuming this code is in the Tour class, we now can call fullprice_for_tour* on the Tour class.  This will then get the fullprice object inside this instance and call the for_tour method on it.

tour.fullprice_for_tour would be the same as tour.fullprice.for_tour.

* I’ve changed the method name to fullprice in order to make the code example simpler.

This kind of code is clever.  But it stymies discoverability again.  When I search for the method fullprice_for_tour I will be unable to find any definition of it anywhere.  I now need to investigate the Tour class file in order to determine that there is a method_missing handler, and work out that actually we are calling fullprice on that class and for_tour on the FullPrice class.  Now I can find the code.

The simple model of searching for the implementation is broken by this coding style.  Searching now becomes an iterative process when nothing comes up.  Which takes longer.

And then there are Rails delegates

In Rails you can add to the Tour class

  delegate :for_tour, to: :full_price

which enables

  tour.for_tour

to be the same as tour.full_price.for_tour

You can even add prefixes

  delegate :for_tour, to: :full_price, prefix: :full_price

which now enables

  tour.full_price_for_tour

to call tour.full_price.for_tour

This saves a developer from writing

  def full_price_for_tour
    full_price.for_tour
  end

in the Tour class.

We save writing a method definition.  But the discoverability is hurt – particularly when the prefix is used.  We now have to do multiple different types of searches in order to discover where full_price_for_tour is defined.  And we need to remember to do that.  And as determined, there could be multiple different ways in which the method could be defined dynamically.

A Hypothesis

The cost of discovery should be at least N times lower than the cost to write the code.  Where N is the total number of times the code is to be viewed and understood up until it is deleted.

I would hypothesise that the benefit of not having to write a trivial method definition makes the discoverability of the method take at least twice as long.  In general my first guess will be wrong.  I have to guess at least once more – looking for delegates.  But then again, it might be another dynamic way, so I might need to keep on guessing.

The design choice of coding this way results in a codebase that on average takes longer to discover things in.  Which means over time, software will take longer and longer to be delivered.  As compared to the constant cost at the time of typing a little more.

The constant cost of typing the method definition occurs once – when the developer writes it.  The cost of discovery occurs every time a developer needs to understand where the code is defined or from where it is called.

So is dynamic meta programming ever justified?

For the majority of developers, the core type of work is business applications that mostly do CRUD operations.  Use cases are driven by actual requirements.  Actual requirements are concretely defined.  They should have concrete tests that define them.  Using dynamic meta programming is almost never required.

Sometimes the code is doing the same thing behind those concrete interfaces.  The code may want to be refactored to take advantage of dynamic techniques to reduce duplication and expose a new level of abstraction.  This can be valuable when things are in fact the same.  If the abstraction makes the system easier to change, this is good.  But these changes should be done beneath the concrete definitions of what the system does.  The system is a set of concrete interfaces and use cases that have concrete tests.  That is what allows us to refactor to a more abstract design below the covers.  As the underlying code becomes more abstract, the external interface and the tests calling the interface remain specific.  The abstraction should not be the exposed interface of your average business application.  The abstraction should not make it harder to discover how the system works.

Observations

Many developers value locally optimizing the time that they save writing code.   At the same time, they ignore the amount of time they cause someone else to waste when attempting to work out the implementation at a later date. Most code is write once, read many.  Optimising for discoverability and understanding is more useful on your average business application than optimizing for the speed at which you can take on the next story.  Optimising for speed to the next story now will result in slowing down later due to spending time discovering how to change the code in order to implement the new story.

I value discoverability.  Having worked on many large code bases – finding stuff needs to be as easy as possible.  I understand others may value terseness more.  Design is always a trade-off.  Understanding what is being traded-off is important.  I don’t consider using meta programming, to reduce lines of code that I need to write, more important than being able to discover and understand that code quickly and reliably later.

If your team uses code like the Rails delegate everywhere in their code – then everyone already knows that all searches to discover a method’s usage or implementation should take that into account. Everyone will be doing it and perhaps that is fine – despite increasing the complexity of that search.  The importance here is consistency and providing an element of least surprise.

If a codebase sometimes uses magic – method_missing, delegates, etc – and sometimes does not, then it becomes more of a guessing game when to search for them and when not to.  That is a real cost to maintaining a codebase.

If I haven’t found the code in my search – is that because it isn’t there or is it because it is using some other magical way in order to be declared?

Don’t use dynamic meta programming unless it is really useful.  Most things are cleaner, clearer and easier to change without meta programming.

If you’re breaking the paradigm, use something else to mitigate the loss.  In the case of Active Record, using the annotate gem to help discoverability mitigates the dynamic implementation that makes discovery of the methods harder.

Think!

Think about discoverability.  Think about the cost of making discoverability harder.  Is there something that can be done to mitigate the cost of this design choice?

All design choices are choices.  Weigh up the pros and cons for yourself and with your team.  Discoverability is just one facet of a good code design, but all too often it isn’t even a factor in the thought process.