Groovy Dependencies, Grape, and Firewalls

Groovy is a powerful and fun language. One of the nice features, particularly useful for internal tools,  is the ability to deliver scripts that are self contained with the only pre-requisite being that you have groovy installed. One of the tools that enables this is the Grape dependency management mechanism. Using Grape, rather than delivering a zip file with all dependencies, or requiring that modules be installed as part of the global system installation, you can add a line like this to your script:

@Grab(group='org.springframework', module='spring', version='2.5.6')

This will grap the dependency into a local repository if it is not present already. It makes the startup time for the first run of a script a bit slower, but it also greatly simplifies the delivery of short, script-based tools.

If you are in production environment behind a firewall, you might find that that this mechanism doesn't work because you can't access the public repositories. You can address this by adding an annotation to the file to add your repository to the list

@GrabResolver(name='restlet', root='http://maven.mycompany.com/proxy')

When I first tried this, it seemed to take a very long time for the script to run, with no output.

To diagnose the problem, I needed to have the GrabResolver tell me what it was doing. Looking at the code the way to do this was to set the ivy.message.logger.level system property.

My command line to get detailed output was:

groovy -Divy.message.logger.level=4 Script.groovy

This gave me enough output to see that Grape was looking in the local repository last after timing out when trying the standard repositories. The easiest fix I found for this was create a $HOME/.groovy/GrapeConfig.xml file as described in the Grape documentation. The new local config only has one ibiblio repository element, which is the company repository manager.

<ivysettings>
  <settings defaultresolver="downloadGrapes">
  <resolvers>
    <chain name="downloadGrapes">
      <filesystem name="cachedGrapes">
        <ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml">
        <artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]">
      </artifact></ivy></filesystem>
      &lt;ibiblio m2compatible="true" name="proxy-repository" root="http://maven.mycompany.com/proxy/">
    </ibiblio></chain>
  </resolvers>
</settings>
</ivysettings>

This sped things up significantly.

There may be another way to address the problem, and I welcome feedback. But since this wasn't obvious from the documentation, I thought it might be worth sharing.

Note, this is based on Groovy 2.0.2.

Problem Solving: Deadlines and Context

One of the more difficult challenges people and teams have in the face of deadline pressure is taking time to consider how to approach a problem rather than just diving in with a solution approach that you know will work.

In college, when I was taking a devices and circuits class,  I found myself stuck on a problem on the first problem set of the semester.  I asked an upperclassman for help and we determined that the problem could be solved by setting up a system of something like 12 equations, and grinding through the process of solving it. His solution was correct of course, but I wondered if this approach might be more complex than it needed to be given that it was  for one of a number of problems in the first problem set of the course, and given that the theme of the first few classes was more conceptual than about doing calculations .

When I later saw the solution to the problem it turned out that by making some simplifying assumptions, the 12 equations reduced to 2, and the solution was quite simple and quick. The lesson that the problem was mean to teach was not how to solve linear equations, but about how to understand when how to use a simple model to understand a complex circuit. The direct, labor intensive approach ignored what the problem was meant to teach.

I don't recall much about the specifics of the problem, but one thing I learned from that experience is that while working harder will often (though not always!) give you a correct solution,  it's always good to think about more than one approach before diving in and solving a problem. 


Sometimes just working harder is the right thing to do. But if you need to get past a deadline, before diving in to solving a problem it's good to think about whether you're doing more work than the situation merits, and that, perhaps, you are overlooking a simpler, quicker,  solution. 

Do we care whether it's Software Engineering or Craft?

Much like a tenet of agile software development is that "planning is more important than the plan," there are some questions about software development that are useful to explore, even if you can't suggest a good answer. One of these these is whether what we, as software developers do, is (or can be) engineering.

I was talking with a colleague about a comment someone made about a post about lessons that software engineers can learn from artists. In this comment the poster raised an issue that comes up a lot when someone uses the phrase "software engineer." The question was, in essence:

Is what we do when we build software "engineering" or "craft?"

My colleague, who has a background in Naval Engineering and who is engaged to an Electrical Engineer (who designs and builds hardware) suggested that the difference is that one rule of thumb might be:

It's engineering if you need to use calculation (about physical constraints).

An example was that to build a bridge you need to do calculations to determine if the bridge, as designed, will support the load you want to support. Many of the "classical" engineering disciplines (Mechanical, structural, electrical) fit that criterion. But by that definition, some things which most would agree that are crafts, such as carpentry, involve calculations. So there is probably more to it that that.

On the other hand, I don't think that all work that is in a discipline that we might call "engineering" is engineering. A good friend designs test equipment for a living. When he builds a sensor to measure the performance of his home heating system is he doing engineering, or is that activity more like a hobby or craft?

As I think of the question, and the frequency at which it comes up, I wonder:

Why does it matter whether we are software engineers or software craftsmen?

Engineering, to me, implies a certain level of discipline are repeatable process, and all of the classic engineering disciplines are grounded in physical laws that we are fairly confident in for the scales at which they apply.

If the "software engineering" question is important to you, take a minute to understand why you are asking the question, as the reason for asking may affect the answer.  You can bring discipline to building something you regardless of whether or not you are engineering. I don't think that the question of how (or whether) to make software into an engineering discipline is an unimportant one. But I do think that it is often asked in a vacuum, much like the an attempt at a user story that is missing the "so that..." clause.

To me, as a professional, it's more important that I bring discipline to what I do. I want to build useful, quality, software where the definition of "quality" that is often most appropriate is Jerry Weinberg's from Quality Software Management: Systems Thinking:

Quality is Value to Some Person.

It might be fair to say that we're not always "crafting" or "engineering" software when we sit down at an IDE and program. Sometimes we're just coding (or hacking, or programming). But if you (and your team) are committed to building software that meets your customers needs, and you are always working to learn how to work better, you're doing a good thing. And you don't need to call it engineering to acknowledge that.

Refactoring

Frank Buschmann wrote a 2-part article in the July/August and September/October issues of IEEE Software that covered an important aspect of software development: refactoring.

Yes,  you'll hear the word "refactoring" spoken often in any development team, but most developers rarely discuss what refactoring is, and whether they are refactoring or doing something else, like "reengineering" or even re-writing.

I won't repeat what Frank said, as he said it quite well,  but as I read the article I realized that there are some points that get lost in many teams and are worth emphasizing:

• Refactoring does not mean "any kind of changing." It means changing the structure of existing code to preserve functionality. In principle you are making the code "better."
• To refactor you need a way to validate that you have preserved functionality. Automated tests are probably the best way to do this. I suggest that you can't refactor without having a test plan in place. And from a practical perspective, this means automated tests.
• Refactoring is a way to improve code quality, but like other things a development teams spends time on,  you you should refactor when it provides business value. 

Being precise about when a change is refactoring and when it's reengineering, a rewrite, or simply changing code to add a feature helps set expectations with stakeholders and helps you as a developer better understand when you'll be done. And by thinking in terms of refactoring continuously you can deliver code more quickly, and avoid the need for reengineering or re-writing, and better be able to evaluate functionality with incremental delivery.

Refactoring is good. Not all good coding is refactoring.

Specialization, Generalization, and Effectiveness in Software Teams: Clinical Metaphors

I was thinking about the relative value to a team of a developer with specific skills (say UI development) versus adding someone who was more of an end-to-end developer. Two stories about medical practice that provided some insight into the question.

I recently read Better, Atul Gawande's book about improvement in medical professions. In this book he relates a story of a clinic in rural India that is poorly staffed and funded, and where doctors manage to successfully perform procedures that are outside the realm of their training. They are able to practice these skills effectively and saved lives.

A story on This American Life, discussed a doctor  in Kermit, Texas who practices in areas beyond his stated training, with poor results. In the end nurses who worked with the doctor filed complaints against him for mistreating patients. This doctor worked alone, and collected information from questionable sources, and ignored the availability of better qualified resource near by.

In both situations doctors practiced beyond their training, yet in one case this improved care, in the other it degraded care.  Two of the key differences seemed to be that the doctors in India frequently met to discuss each other's cases and learn from each other, and  the doctors in india worked beyond their area of specialization in order to keep the clinic functioning and performing useful work. There was no way to wait for a specialist to do the work, and still keep the patient alive.  Gawande credits the camaraderie of the group in India as well as their daily discussions of their cases, their decisions, and why they made those decisions, for their ability to improve their range of practice.

Even though the dynamics in software teams differ from those in a hospital, there are lessons from these stories that software teams can apply to better develop teams of generalizing specialists, and more effective agile developers:

• An ethic of continual learning, both from outside resources and each other can help a team of generalizing specialists to be effective; Having the ability to work across a stack isn't as important as a willingness and ability to learn how to fill gaps in knowledge.
• The flow of the project is  important  to consider when deciding who should work on a task, consider the overall flow of the project. If the specialist is available, there you may as well have the "expert" work on the task (after the team agrees on an approach ). But if waiting for the specialist would delay the project, and there are other people on the team who don't have a full backlog of work, consider having someone else work on it. The specialist can provide advice, and perhaps improve on the work later. But you will have functional software sooner.
• It's important to take time to review and discuss decisions as a cross-functional team to understand what you did, and what you can do better next time.

So it seems like it's best add to a person to a team who has a skill that the team might be weak in, but who enjoy working on the whole application when the team thinks it makes sense. Self-organizing, cross-functional teams are a central part of agile practice, and the combination is important. Just having cross functional people doesn't work; they need to work as part of a team working towards a common goal.

Book Review: Modular Java

I recently read  Craig Walls' book Modular Java: Creating Flexible Applications with Osgi and Spring (Pragmatic Programmers). This book is a very detailed tutorial that walks you through setting up an application using OSGI and Spring with the help of Maven as a build tool. If you aren't familiar with any of these technologies, this book will get you started, and quickly have you feeling like you have a basic grasp of the concepts and technologies.

You'll finish the book with a desire to learn more about the technologies, and understand the power of modular applications.  As a tutorial, this is an excellent book. This is not an general guide to how to design with OSGI.  There is some background on the frameworks, and some explanation of technologies, and also pointers to sources for more information, but this book is all about learning by building. If you pick up the book hoping to learn any details about the how and why of OSGI and Spring, two useful technologies, you might be disappointed.  But if you like learning by doing, and working with running code, you'll enjoy and value this book.

97 Things Every Programmer Should Know

This past week 97 Things Every Programmer Should Know was made available to the public. This project was driven by Kevlin Henney, who I know from the early Pattern Languages of Programming conferences, . The list of contributors contains many familiar names, and I'm honored to be among them.

My contributions are about the agility, deployment, and their intersection. They include:

• Own (and Refactor) the Build
• Deploy Early and Often
• Understand the Principles behind Practices, and
• Acknowledge (and Learn From) Failures
  

This project has contributions from many really interesting and insightful people. Maybe you already know everything on the lists, but you might get a fresh perspective by reading the contributions. You are very likely to start thinking.

On a related note, this past week, there was a conversation in my office between one of my colleagues who is a parent of a toddler, and one who isn't about why one works so hard to sound excited when talking to children about tasks.

You might say in an excited tone:

"Let's brush our teeth! It will be fun!"
instead of
"You need to brush your teeth because it's good for you..."

Being the proud (some who know me IRL might say too proud) parent of a 2-year, 9-month old boy, this got me thinking about why parents do this, and why it works so well. I suspect that the reasons are two-fold:

• Toddlers don't (yet) get the idea of consequences, but they get the idea of "fun" so framing something as fun and exciting is just the path of least resistance.
• Toddlers are wired to explore and learn, and every new thing is fun!

I'm realizing that, while as professionals, we need to do things for purely practical reasons learning about how to work better can (and perhaps should) be fun too. Seeing how toddlers learn makes me wonder how much of that joy of discovery we give up as we grow, and whether we need to lose it.

So as you read through the contributions in 97 Things, try to find something new and learn about it not just because it's something you should know, but because you enjoy programming, and because learning new things is fun!