DaedTech

Stories about Software

By

Improving Readability using Moq

While I’ve been switching a lot of my new projects over to using JustMock, as explained here, I still have plenty of existing projects with Moq, and I still like Moq as a tool. One of the things that I have struggled with in Moq that JustMock provides a nice solution to is the cognitive dissonance that arises when thinking about how to reason about your test doubles.

In some situations, such as when you’re injecting the test doubles into your class under test, you want them to appear as if they were just another run of the mill instance of whatever interface/base you’re passing. But at other times you want to configure the mock, and then you want to think of them as actual mock objects. One solution to that is to use Mock.Get() as I described here. Doing that, you can store a reference to the object as if it were any other object but access it using Mock.Get() when you want to do things to its setup:

[gist id=”4998372″]

That’s all well and good, but, perfectionist that I am, I got really tired of all of that Mock.Get() ceremony. It makes lines pointlessly longer, and I think really detracts from the readability of the test classes. So I borrowed an idea from JustMock with its “Helpers” namespace and created a series of extension methods. Some examples are shown here.

[gist id=”4998277″]

These extension methods allow me to alter my test to look like this:

[gist id=”4998488″]

Now there’s no need to switch context, so to speak, between mock object and simple dependency. You always have a simple dependency that just so happens to have some extension methods that you can use for configuration. No need to keep things around as Mock<T> and call the Object property, and no need for all of the Mock.Get().

Of course, there are caveats. This might already exist somewhere and I’m out of the loop. There might be issues with this that I haven’t yet hit, and there is the debugging indirection. And finally, you could theoretically have namespace collisions, though if you’re making methods called “Setup” and “Verify” on your classes that take expression trees and multiple generic parameters, I’d say you have yourself a corner case there, buddy. But all that aside, hopefully you find this useful–or at least a nudge in a direction toward making your own tests a bit more readable or easy to work with.

By

The Way We Write Code is Stupid: Source Code Files Considered Harmful

Order Doesn’t Matter

Please pardon the loaded phrasing in the title, but that’s how the message came to me from my subconscious brain: bluntly and without ceremony. I was doing a bit of work in Apex, the object-oriented language specific to Salesforce.com, and it occurred to me that I had no idea what idiomatic Apex looked like. (I still don’t.) In C++, the convention (last time I was using it much, anyway) is to first define public members in class headers and then the private members at the bottom. In C#, this is inverted. I’ve seen arguments of all sorts as to which approach is better and why. Declaring them at the top makes sense since the first thing you want to see in the class is what its state will consist of, but declaring the public stuff at the top makes sense since that’s what consumers will interact with and it’s like the above-water part of your code iceberg.

When programming in any of the various programming languages I know, I have this mental cache of what’s preferred in what language. I attempt to ‘speak’ it without an accent. But with Apex, I have no idea what the natives ‘sound’ like, not having seen it in use before. Do I declare instance variables at the bottom or the top? Which is the right way to eat bread: butter side up or butter side down? I started googling to see what the ‘best practice’ was for Apex when the buzzing in my subconscious reached some kind of protesting critical mass and morphed into a loud, clear message: “this is completely stupid.”

I went home for the day at that point–it was late anyway–and wondered what had prompted this visceral objection. I mean, it obviously didn’t matter from a compiled code perspective whether instance variables or public methods come first, but it’s pretty well established and encouraged by people as accomplished and prominent as “Uncle” Bob Martin that consistency of source code layout matters, if not the layout specifics (paraphrased from my memory of his video series on Clean Coders). I get it. You don’t want members of your team writing code that looks completely different from class to class because that creates maintenance headaches and obscures understanding. So what was my problem?

I didn’t know until the next morning in the shower, where I seem to do my most abstract thinking. I didn’t think it was stupid to try to make my Apex code look like ‘standard’ Apex. I thought it was stupid that I needed to do so at all. I thought it was stupid to waste any time thinking about how to order code elements in this file when the only one whose opinion really matters–the compiler–says, “I don’t care.” Your compiler is trying to tell you something. Order doesn’t matter to it, and you shouldn’t care either.

Use Cases: What OOP Developers Want

But the scope of my sudden, towering indignation wasn’t limited to the fact that I shouldn’t have to care about the order of methods and fields. I also shouldn’t have to care about camel or Pascal casing. I shouldn’t have to care about underscores in front of field names or inside of method names. It shouldn’t matter to me if public methods come before private or how much indentation is the right amount of indentation. Should methods be alphabetized or should they be in some other order? I don’t care! I don’t care about any of this.

Let’s get a little more orderly about this. Here are some questions that I ask frequently when I’m writing source code in an OOP language:

  • What is the public API of this type?
  • What private methods are in the ‘tree’ of this public method?
  • What methods of this type mutate or reference this field?
  • What are the types in this namespace?
  • What are the implementations of this interface in this code base?
  • Let’s see this method and any methods that it overrides.
  • What calls this method?

Here are some questions that I never ask out of actual interest when writing source code.  These I either don’t ask at all or ask in exasperation:

  • What’s the next method in this file?
  • How many line feed characters come before the declaration of this variable?
  • Should I use tabs or spaces?
  • In what region is this field’s declaration?
  • Did the author of this file alphabetize anything in it?
  • Does this source file have Windows or *NIX line break characters?
  • Is this a field or a method or what?

With the first set of questions, I ask them because they’re pieces of information that I want while reasoning about code.  With the second set of questions, they’re things I don’t care about.  I view asking these questions as an annoyance or failure.  Do you notice a common pattern?  I certainly do.  All of the questions whose answers interest me are about code constructs and all the ones that I don’t care about have to do with the storage medium for the code: the file.

But there’s more to the equation here than this simple pattern.  Consider the first set of questions again and ask yourself how many of the conventions that we establish and follow are simply ham-fisted attempts to answer them at a glance because the file layout itself is incapable of doing so.  Organizing public and private separately is a work-around to answer the first question, for example.  Regions in C#, games with variable and method naming, “file” vs “type” view, etc. are all attempts to overcome the fact that files are actually really poor communication media for object-oriented concepts.  Even though compilers are an awful lot different now than they were forty years ago, we still cling to the storage medium for source code best suited to those old compilers.

Not Taking our own Advice

If you think of an ‘application’ written in MS Access, what comes to mind?  How about when you open up an ASP web application and find wizard-generated data sources in the markup, or when you open up a desktop application and find SQL queries right in your code behind?  I bet you think “amateurs wrote this.”  You are filled with contempt for the situation–didn’t anyone stop to think about what would happen if data later comes in some different form?  And what about some kind of validation?  And, what the–ugh… the users are just directly looking at the tables and changing the column order and default sorting every time they look at the data.  Is everyone here daft?  Don’t they realize how ridiculous it is to alter the structure of the actual data store every time someone wants a different ordering of the data?

OldYoungAnd you should see some of the crazy work-arounds and process hacks they have in place. They actually have a scheme where the database records the name of everyone who opens up a table and makes any kind of change so that they can go ask that person why they did it.  And–get this–they actually have this big document that says what the order of columns in the table should be.  And–you can’t make this stuff up–they fight about it regularly and passionately.  Can you believe the developers that made this system and the suckers that use it? I mean, how backward are they?

In case you hadn’t followed along with my not-so-subtle parallel, I’m pointing out that we work this way ourselves even as we look with scorn upon developers who foist this sort of thing on users and users who tolerate it.  This is like when you finally see both women in the painting for the first time–it’s so clear that you’ll never un-see it again.  Why do we argue about where to put fields and methods and how to order things in code files when we refuse to write code that sends users directly into databases, compelling them to bicker over the order of column definition in the same?  RDBMS (or any persistence store) is not an appropriate abstraction for an end user–any end user–whether he understands the abstraction or not.  We don’t demand that users fight, decide that there is some ‘right’ way to order invoices to be printed, and then lock the Invoice table in place accordingly for all time and pain of shaming for violations of an eighty-page invoice standard guideline document.  So why do that to ourselves?  When we’re creating object-oriented code, sequential files, and all of the particular orderings, traversings and renderings thereof are wildly inappropriate abstractions for us.

What’s the Alternative?

Frankly, I don’t know exactly what the alternative is yet, but I think it’s going to be a weird and fun ride trying to figure that out.  My initial, rudimentary thoughts on the matter are that we should use some sort of scheme in which the Code DOM is serialized to disk for storage purposes.  In other words, the domain model of code is that there is something called Project, and it has a collection of Namespace.  Namespace has a collection of Type, which may be Interface, Enum, Struct, Class (for C# anyway–for other OOP languages, it’s not hard to make this leap).  Class has one collection each of Field, Method, Property, Event.  The exact details aren’t overly important, but do you see the potential here?  We’re creating a hierarchical model of code that could be expressed in nested object or relational format.

In other words, we’re creating a domain model entirely independent of any persistence strategy.  Can it be stored in files?  Sure. Bob’s your uncle.  You can serialize these things however you want.  And it’ll need to be written to file in some form or another for the happiness of the compiler (at least at first).  But those files handed over to the compiler are output transforms rather than the lifeblood of development.

Think for a minute of us programmers as users of a system with a proper domain, one or more persistence models, and a service layer.  Really, stop and mull that over for a moment.  Now, go back to the use cases I mentioned earlier and think what this could mean.  Here are some properties of this system:

  1. The basic unit of interaction is going to be the method, and you can request methods with arbitrary properties, with any filtering and any ordering.
  2.  What appears on your screen will probably be one or more methods (though this would be extremely flexible).
  3. It’s unlikely that you’d ever be interested in “show me everything in this type.”  Why would you?  The only reason we do this now is that editing text files is what we’re accustomed to doing.
  4. Tracing execution paths through code would be much easier and more visual and schemes that look like Java’s “code bubbles” would be trivial to create and manipulate.
  5. Most arguments over code standards simply disappear as users can configure IDE preferences such as “prepend underscores to all field variables you show me,” “show me everything in camel casing,” and, “always sort results in reverse alphabetical order.”
  6. Arbitrary methods from the same or different types could be grouped together in ad-hoc fashion on the screen for analysis or debugging purposes.
  7. Version/change control could occur at the method or even statement level, allowing expression of “let’s see all changes to this method” or “let’s see who made a change to this namespace” rather than “let’s see who changed this file.”
  8. Relying on IDE plugins to “hop” to places in the code automatically for things like “show all references” goes away in favor of an expressive querying syntax ala NDepend’s “code query language.”
  9. New domain model allows baked-in refactoring concepts and makes operations like “get rid of dead code” easier or trivial, in some cases.

Longer Reaching Impact

If things were to go in this direction, I believe that it would have a profound impact not just on development process but also on the character and quality of object oriented code that is written in general.  The inherently sequential nature of files and the way that people reason about file parsing, I believe, lends to or at least favors the dogged persistence of procedural approaches to object oriented programming (static methods, global state, casting, etc.).  I think that the following trends would take shape:

  1. Smaller methods.  If popping up methods one at a time or in small groups becomes the norm, having to scroll to see and understand a method will become an anomaly, and people will optimize to avoid it.
  2. Less complexity in classes.  With code operations subject to a validation of sorts, it’d be fairly easy to incorporate a setting that warns users if they’re adding the tenth or twentieth or whatever method to a class.  In extreme cases, it could even be disallowed (and not through the honor system or ex post facto at review or check in–you couldn’t do it in the first place).
  3. Better conformance to Single Responsibility Principle (SRP).  Eliminating the natural barrier of “I don’t want to add a new file to source control” makes people less likely awkwardly to wedge methods into classes in which they do not belong.
  4. Better cohesion.  It becomes easy to look for fields hardly used in a type or clusters of use within a type that could be separated easily into multiple types.
  5. Better test coverage.  Not only is this a natural consequence of the other items in this list, but it would also be possible to define “meta-data” to allow linking of code items and tests.

What’s Next?

Well, the first things that I need to establish is that this doesn’t already exist somewhere in the works and that I’m not a complete lunatic malcontent.  I’d like to get some feedback on this idea in general.  The people to whom I’ve explained a bit so far seem to find the concept a bit far-fetched but somewhat intriguing.

I’d say the next step, assuming that this passes the sanity check would be perhaps to draw up a white paper discussing some implementation/design strategies with pros and cons in a bit more detail.  There are certainly threats to validity to be worked out such as the specifics of interaction with the compiler, the necessarily radical change to source control approaches, the performance overhead of performing code transforms instead of just reading a file directly into memory, etc.  But off the top of my head, I view these things more as fascinating challenges than problems.

In parallel, I’d like to invite anyone who is at all interested in this idea to drop me an email or send me a tweet.  If there are others that feel the way I do, I think it’d be really cool to get something up on Github and maybe start brainstorming some initial work tasks or exploratory POCs for feasibility studies.  Also feel free to plus-like-tweet-whatever to others if you think they might be interested.

In conclusion I’ll just say that I feel like I’d really like to see this gain traction and that I’d probably ratchet this right to the top of my side projects list if people are interested (this being a bit large in scope for me alone in my spare time).  Now whenever I find myself editing source files in an IDE I feel like a bit of a barbarian, and I really don’t think we shouldn’t have to tolerate this state of affairs anymore.  Productivity tools designed to hide the file nature of our source code from us help, but they’re band-aids when we need disinfectants, antibiotics, and stitches.  I don’t know about you, but I’m ready to start writing my object-oriented code using an IDE paradigm that doesn’t support GOTO Line as if we were banging out QBasic in 1986.

By

Nuget: Stop Copying and Pasting your Utils.cs Class Everywhere

I was setting up to give a presentation the other day when it occurred to me that Nuget was the perfect tool for my needs. For a bit of background, I consider it of the utmost importance to tell a story while you present. Things I’m not fond of in presentations include lots of slides without any demonstration and snapshots of code bases in various stages of done. I prefer a presentation where you start with nothing (or with whatever your audience has) and you get to some endpoint, by hook or by crook. And Nuget is an excellent candidate for whichever of “hook” and “crook” means “cheating.” You can get to where you’re going by altering your solution on the fly as you go, but without all of the umming and hawing of “okay, now I’ll right click and add an assembly reference and, oh, darnit, what was that fully qualified path again, I think–oops, no not that.”

In general, I’ve decided to set up an internal Nuget feed for some easier ala carte development, so this all dovetails pretty nicely. It made me stop and think that, given how versatile and helpful a tool this is, I should probably document it for anyone just getting started. I’m going to take you through creating the simplest imaginable Nuget package but with the caveat that this adds a file to your solution (as opposed to other tutorials you’ll see that often involve marshaling massive amounts of assembly references or something). So, here are the steps (on Windows 7 with VS2012):

  1. Go to codeplex and download the package explorer tool.
  2. Install the package explorer, launch it, and choose “Create a New Package”:

    NugetCreateNew

  3. In the top left, click the “Edit Metadata” icon as shown here:

    EditMetadata

  4. Fill out the metadata, ensuring that you fill things in for the bold (required) fields:

    ActualMetadata

  5. Click the green check mark to exit the metadata screen and then right-click in the “Package Contents” pane. Select “Add Content Folder.” The “Content” folder is a structure of files that will mimic what Nuget puts in your solution.

    NewContent

  6. Now, right click on the “Content” folder and select “Add New File.” Name the file “Readme.txt” and add a line of text to it saying “Hello.” Click the save icon and then the “back” arrow next to it.
  7. Now go to the file menu at the top and select “Save,” which will prompt you for a file location. Choose the desktop and keep the default naming of the file with the “nupkg” extension. Close the package explorer.
  8. Create a new solution in Visual Studio into which we’re going to import the Nuget package.
  9. Go to Tools->Library Package Manager->Manage Packages for Solution to launch the Nuget GUI and click “Settings” in the bottom right because we’re going to add the desktop as a Nuget feed:

    NugetGui

  10. In this screen, click the plus icon (pointed to by the arrow) and then give the package a name and enter the file path for the desktop by way of configuration:

    AddPackageSource

  11. Click “Ok” and you should see your new Nuget feed listed underneath the official package source. If you click it, you should see your test package there:

    TestPackage

  12. Click “Install” and check out what happens:

    NugetSuccess

Observe that source (or text or whatever) files are installed to your solution as if they were MSI installs. You can install them, uninstall them, and update them all using a GUI. Source code is thus turned into a deliverable that you can consume without manual propagation of the file or some kind of project or library include of some “CommonUtils” assembly somewhere. This is a very elegant solution.

It shouldn’t take a ton of extrapolation to see how full of win this is in general, especially if you create a lot of new projects such as a consulting shop or some kind of generalized support department. Rather than a tribal-knowledge mess, you can create an internal Nuget feed and setup a sort of code bazaar where people publish things they think are useful; tag them with helpful descriptors; and document them, allowing coworkers to consume the packages. People are responsible for maintaining and helping with packages that they’ve written. Have two people do the same thing? No worries–let the free market sort it out in terms of which version is more popular. While that may initially seem like a waste, the group is leveraging competition to improve design (which I consider to be an intriguing subject, unlike the commonly embraced anti-pattern, “design by committee,” in which people leverage cooperation to worsen design).

But even absent any broader concerns, why not create a little Nuget feed for yourself, stored in your Dropbox or on your laptop or whatever? At the very least, it’s a handy way to make note of potentially useful things you’ve written, polish them a little, and save them for later. Then, when you need them, they’re a click away instead of a much more imposing “oh, gosh, what was that one thing I did that one time where I had a class kinda-sorta like this one…” away.

By

API Lost Opportunity: A Case Study

I’m aware that recently there’s been some brouhaha over criticism/trashing of open-source code and the Software Craftsmanship movement, and I’d like to continue my longstanding tradition of learning what I can from things without involving myself in any drama. Toward that end, I’d like to critique without criticizing, or, to put it another way, I’d like to couch some issues I have scan0001with an API as things that could be better rather than going on some kind of rant about what I don’t like. I’m hoping that I can turn the mild frustration I’m having as I use an API into being a better API writer myself. Perhaps you can pick up a few tidbits from this as well.

Recently I’ve been writing some code that consumes the Microsoft CRM API using its SDK. To start off with the good, I’d say that API is fairly powerful and flexible–it gives you access to the guts of what’s in the CRM system without making you write SQL or do any other specific persistence modeling. Another impressive facet is that it can handle the addition of arbitrary schema to the database without any recompiling of client code. This is done, in essence, by using generalized table-like objects in memory to model actual tables. Properties are handled as a collection of key-value string pairs, effectively implementing a dynamic-typing-like scheme. A final piece of niceness is that the API revolves around a service interface which makes it a lot nicer than many things for mocking when you test.

(If you so choose, you can use a code generator to hit your CRM install and spit out strongly-typed objects, but this is an extra feature about which it’s not trivial to find examples or information. It also spits out a file that’s a whopping 100,000+ lines, so caveat emptor.)

But that flexibility comes with a price, and that price is a rather cumbersome API. Here are a couple of places where I believe the API fails to some degree or another and where I think improvement would be possible, even as the flexible scheme and other good points are preserved.

Upside-Down Polymorphism

The API that I’m working with resides around a series of calls with the following basic paradigm:

There is a service that accepts very generic requests as arguments and returns very generic responses as return values. The first component of this is nice (although it makes argument capture during mocking suck, but that’s hardly a serious gripe). You can execute any number of queries and presumably even create your own, provided they supply what the base wants and provided there isn’t some giant switch statement on the other side downcasting these queries (and I think there might be, based on the design).

The second component of the paradigm is, frankly, just awful. The “VeryAbstractBaseResponse” provides absolutely no useful functionality, so you have to downcast it to something actually useful in order to evaluate the response. This is not big-boy polymorphism, and it’s not even adequate basic design. In order to get polymorphism right, the base type would need to be loaded up with methods so that you never need to downcast and can still get all of the child functionality out of it. In order not to foist a bad design on clients, it has to dispense with needing to base your response type on the request type. Here’s what I mean:

Think about what this means. In order to get a meaningful response, I need to keep track of the type of request that I created. Then I have to have a knowledge of what responses match what requests on their side, which is perhaps available in some horrible document somewhere or perhaps is only knowable through trial/error/guess. This system is a missed opportunity for polymorphism even as it pretends otherwise. Polymorphism means that I can ask for something and not care about exactly what kind of something I get back. This scheme perverts that–I ask for something, and not only must I care what kind of something it is, but I also have to guess wildly or fail with runtime exceptions.

The lesson?

Do consumers of your API a favor. Be as general as possible when accepting arguments and as specific as possible when returning them. Think of it this way to help remember. If you have a method that takes type “object” and can actually do something meaningful with it, that method is going to be incredibly useful to anyone and everyone. If you have a method that returns type object, it’s going to be pretty much useless to anyone who doesn’t know how the method works internally. Why? Because they’re going to have to have an inappropriate knowledge of your workings in order to know what this thing is they’re getting back, cast it appropriately, and use it.

Hierarchical Data Transfer Objects

With this interface, the return value polymorphism issues might be livable if not for the fact that these request/response objects are also reminiscent of the worst in OOP, reminiscent of late ’90s, nesting-happy Java. What does a response consist of? Well, a response has a collection of entities and a collection of results, so it’s not immediately clear which path to travel, but if you guess right and query for entities, you’re just getting started. You see, entities isn’t an enumeration–it’s something called “EntityCollection” which brings just about nothing to the table except to provide the name of the entities. Ha ha–you probably wanted the actual entities, you fool. Well, you have to call response.EntityCollection.Entities to get that. And that’s an enumeration, but buried under some custom type called DataCollection whose purpose isn’t immediately clear, but it does implemented IEnumerable so now you can get at your entity with Linq. So, this gets you an entity:

But you’re quite naieve if you thought that would get you anything useful. If you want to know about a property on the entity, you need to look in its Attributes collection which, in a bit of deja vu, is an AttributeCollection. Luckily, this one doesn’t make you pick through it because it inherits from this DataCollection thing, meaning that you can actually get a value by doing this:

D’oh! It’s an object. So after getting your value from the end of that train wreck, you still have to cast it.

The lesson?

Don’t force Law of Demeter violations on your clients. Provide us with a way of getting what we want without picking our way through some throwback, soul-crushing object hierarchy. Would you mail someone a package containing a series of Russian nesting dolls that, when you got to the middle, had only a key to a locker somewhere that the person would have to call and ask you about? The answer is, “only if you hated that person.” So please, don’t do this to consumers of your API. We’re human beings!

Unit Tests and Writing APIs

Not to bang the drum incessantly, but I can’t help but speculate that the authors of this API do not write tests, much less practice TDD. A lot of the pain that clients will endure is brought out immediately when you start trying to test against this API. The tests are quickly littered with casts, as operators, and setup ceremony for building objects with a nesting structure four or five references deep. If I’m writing code, that sort of pain in tests is a signal to me that my design is starting to suck and that I should think of my clients.

I’ll close by saying that it’s all well and good to remember the lessons of this post and probably dozens of others besides, but there’s no better way to evaluate your API than to become a client, which is best done with unit tests. On a micro-level, you’re dog-fooding when you do that. It then becomes easy to figure out if you’re inflicting pain on others, since you’ll be the first to feel it.

By

Practical Programmer Math: Bit Arithmetic

The “Why This Should Matter to You” Story

Imagine that you’re on a phone interview and things are going pretty well. You’ve fielded some gotcha trivia about whether such and such code would compile and what default parameters are and whatnot. After explaining the ins and outs of the strategy pattern and recursive methods, you’re starting to feel pretty good about yourself when, suddenly, the record skips. “Uh, excuse me?” you stammer.

“Why do integers roll over to negative when you loop all the way up to and past their max values?”

You had always just assumed that this was like an old Nintendo game where, naturally, when you completely leave the right side of the screen you simply appear on the left side. “Uh… they just do…? Pass?”

“No problem–let’s move on. Why do unsigned and signed integers occupy the same number of bits even though signed integers have more information?”

“Oh, they have more information? I know that unsigned integers can’t be negative!”

“Yes, well, okay. Let’s try this: how would you multiply a number by two without using the * operator?”

“Well,” you begin, “I’d–”

“And without adding it to itself.”

“Oh. Huh. I dunno.”

The interviewer sighs a little and says, “okay, so what can you tell me about the way numbers are represented as bits and bytes?”

Remembering the last practical math post, you talk about how you know that bits are 1 or 0 and that 8 of these bits makes up a byte. You talk about how bits can be and-ed and or-ed like booleans, and you offer hopefully that you even know an interesting bit of trivia in that 1K is not actually 1,000 bytes. The interviewer nods absently, and looking back later, you have no trouble pinpointing where things went off the rails. In spite of you knowing a bit about how information is represented at the lowest level, you realize you’re lacking some additional basics as to how this bit and bite stuff works.

You wrap up the interview and when you ask what the next steps are, the interviewer says, “don’t call us–we’ll call you.”

Math Background

First of all, you may want to go back and brush up on some previous posts in this series.

Let’s start with the simplest thing first: bit shifting. To help you understand bit-shifting, I’m going to introduce the concept of “bit significance.” Bit significance is a way of categorizing the “bits” in a binary number from most significant (the “most significant bit” or MSB) to least significant (the “least significant bit” or “LSB”). This is really just a fancy of way of describing the bits that correspond to the largest power of two from greatest to smallest. For example, the number six is written as 0x110, and we would say that the first 1 is the most significant; the middle one the next most; and 0 the least since they correspond to the 4, 2, and 1 values, respectively.

This may seem like a definition so basic as to be pointless, but it’s important to remember that not all architectures necessarily interpret bits from left to right. After all, reading “110” as 6 is rather arbitrary–we could just as easily read it as 3 if we read it the other way. This is reminiscent of how some Middle Eastern written languages are interpreted from right to left. So just to be clear, for the remainder of this post, the MSB will be the leftmost bit and the LSB will be the rightmost.

Bit shifting, often denoted in programming languages by the “>>” and “<<" operators, is the simple procedure of adding a zero to one end or the other of a binary sequence. For example, consider the following code:

What you would wind up with for y is 0x0110 or 6. Notice how there's now an extra 0 on the right and this results in the value being multiplied by two. If we had bit-shifted by 2 instead of 1, we'd have 0x1100 or 12, which is 3 x 4. If you bit-shift the other way, you get back to where you started:

Here, y winds up being 0x0011 or 3 again. Performing a subsequent shift would result in 0x0001 or 1. When you divide unevenly by two, the algorithm 'rounds' down. Another and perhaps easier way to understand this shifting concept is to realize it's actually pretty familiar with the decimal numbers that you're accustomed to. For example, if you "decimal shifted" the number 14 left, you'd have 140 and if you did it to the right, you'd wind up with 1 (rounding down from 1.4). Decimal shifting, as anyone who has "misplaced a zero" knows, is really just a question of multiplying and dividing by powers of 10. Bit-shifting is the same thing, but with powers of two.

Now that you understand a bit about bit significance and the concept of shifting, let's tackle the problem of wrap-around and integer overflow. In dealing with binary numbers so far, we've dealt exclusively with positive integers and zero, meaning that we haven't dealt with negative numbers or non-integers like decimals. The latter will be fodder for a future post on floating point arithmetic, but let's consider how to represent a negative number in binary.

PhoneInterviewIf it were simply a matter of writing it out, that’d be easy. We could simply write -0x11 for -3, the way we do on paper. But when dealing with actual bits on a computer, we don’t have that luxury. We have to signify negativity somehow using the same ones and zeroes that we use for representing the number. Given that negative/positive is an either/or proposition, the common way to do this is with a bit–specifically, the MSB. This lets you represent negativity, but it necessarily cuts in half the count of numbers that you can represent.

Think of a byte, which is 8 bits. Using all 8 bits, you can represent 2^8 = 256 numbers (0 through 255). But using 7, you can only represent 2^7 = 128 (0 through 127). Think of the language construct “unsigned” in most programming languages (with “signed” being the default). When you’re making this distinction, you’re telling the compiler whether or not you want to sacrifice one of the bits in your number type to use as the +/- sign.

Great, but how do we get to the rollover issue? Well, the reason for that is a little more complicated and it involves a concept called “two’s complement.” Let’s say that we represented binary numbers by taking the MSB and making it 0 for positive and 1 for negative. That’s probably what you were picturing in the last paragraph, and it’s called “sign magnitude” representation. It’s workable, but it has the awkward side effect of giving us two ways to represent 0. For instance, consider a signed 3 bit scheme where the first bit was the sign and the next two the number. 101 and 001 would represent negative and positive 1, respectively, but what about 0x100 and 0x000. Positive and negative zero? It also requires different mechanics for arithmetic operations depending on whether the number is positive or negative, which is computationally inefficient. And finally, the fact that there are two versions of zero means that a potential number that could be represented is wasted.

Instead, processors use a scheme called “two’s complement.” In this scheme, the MSB still does wind up representing the sign, but only by side effect. The positive numbers are represented the way they would be in the binary numbers you’re used to seeing. So, in a 4 bit number, the largest normal number would be 7: 0x0111. But when you flip that last bit and go to what would be 8 as a signed number, 0x1000, you’re suddenly representing -8. If you keep counting, you get pairs of (0x1001 or -7), (0x1010 or -6) … (0x1111 or -1). So you walk the scheme up to the most positive number, which you arrive at halfway through, and then you suddenly hit the most negative number and start working your way back to zero. Adding 1 to 0x0111, which corresponds to Integer.Max for a 4 bit integer, would result in 0x1000, which corresponds to Integer.Min.

Negative and positive that you’re used to operate as a line stretching out either way to infinity with the two concepts colliding at zero. But in a two’s complement scheme, they act as a wheel, and you “roll over.” The reason that two’s complement behaves this way is a bit more involved, and I may cover it in a smaller future post, but suffice it to say that it makes bit arithmetic operations easier to reason about algorithmically and more elegant. It also gets rid of that pesky double zero problem.

How it Helps You

In the short term all of this helps you answer the phone interview questions, but I don’t think of that as too terribly important since I consider trivia questions during the interview process to be a hiring anti-pattern, often indicative of a department rife with petty oneupsmanship and knowledge hoarding. You’re better off without that job, and, now that you know the answers to the questions, you have the best of all worlds.

On a longer timeline, you’re learning the fundamentals of how data is represented and works in its most core form in computer science, and that opens some nice doors, in general. Superficially, knowing to bit-shift instead of multiply or divide may allow you to optimize in some environments (though I believe that with most modern compilers they can do pretty well optimizing this on their own). But at a deeper level, you’re gaining an understanding of the internal workings of the data types that you’re using.

When data is sent “over the wire” and you’re examining it with some kind of dump tool, it’s coming in streams of bits. Sure there’s an endpoint somewhere re-assembling it into things like unsigned ints and floats and even higher level abstractions. But if those are the lowest levels at which you understand data, it’s going to make programs hard to debug when you’re using a tool that shows you raw data. It may be tedious to interpret things at that level, but in a tight spot it can be a life saver.

Further Reading

  1. A pretty nice guide to bit shifting
  2. Detailed explanation of Two’s Complement
  3. A nice video on Two’s Complement
  4. A comparison of different schemes for representing negative in bits

By

Switch Statements are Like Ants

Switch statements are often (and rightfully, in my opinion) considered to be a code smell. A code smell, if you’ll recall, is a superficial characteristic of code that is often indicative of deeper problems. It’s similar in concept to the term “red flag” for interpersonal relationships. A code smell is like someone you’ve just met asking you to help them move and then getting really angry when you don’t agree to do it. This behavior is not necessarily indicative of deep-seated psychological problems, but it frequently is.

Consequently, the notion that switch statements are a code smell indicates that if you see switch statements in code, there’s a pretty good chance that design tradeoffs with decidedly negative consequences have been made. The reason I say this is that switch statements are often used to simulate polymorphism for those not comfortable with it:

Clearly a better design from an OOP perspective would be an Animal base class/interface and an Attack() method on that, overridable by children/implementers. This design has the advantage of requiring less code, scaling better, and conforming to the Open/Closed principle–if you want to add some other animal later, you just add a new class to the code base and probably tweak a factory method somewhere and you’re done.

This method isn’t really that bad, though, compared to how bad it could be. The design is decidedly procedural, but its consequences aren’t far reaching. From an execution perspective, the switch statement is hidden as an implementation detail. If you were to isolate the console (or wrap and inject it), you could unit test this method pretty easily. It’s ugly and it’s a smell, but it’s sort of like seeing an ant crawling in your house–a little icky and potentially indicative of an infestation, but sometimes life deals you lemons.

But what about this:

Taking the code at face value, this method figures out what the animal’s attack is and returns it, but it does so by invoking a different module for each potential case. As a client of this code, your path of execution can dive into any of five different libraries (and more if this ‘pattern’ is followed for future animals). The fanout here is out of control. Imagine trying to unit test SavageAntthis method or isolate it somehow. Imagine if you need to change something about the logic here. The violence to the code base is profound–you’d be changing execution flow at the module level.

If the first switch state was like an ant in your code house, this one is like an ant with telephone poles for legs, carving a swath of destruction. As with that happening in your house, the best short-term strategy is scrambling to avoid this code and the best long-term strategy is to move to a new application that isn’t a disaster zone.

Please be careful with switch statements that you use. Think of them as ants crawling through your code–ants whose legs can be tiny ant legs or giant tree trunk legs. If they have giant tree trunk legs, then you’d better make sure they’re the entirety of your application–that the “ant” is the brains ala an Ioc container–because those massive swinging legs will level anything that isn’t part of the ant. If they swing tiny legs, then the damage only occurs when they come in droves and infest the application. But either way, it’s helpful to think of switch statements as ants (or millipedes, depending on the number of cases) because this forces you to think of their execution paths as tendrils fanning out through your application and creating couplings and dependencies.

By

The Council of Elders Anti-Pattern

Convene the High Council

Today I’d like to talk about something that I’ve observed on some occasions throughout my career. It sort of builds on the concepts that I’ve touched on in previous writings “How to Keep your Best Programmers” and the Expert Beginner posts, including Bruce Webster’s “Dead Sea Effect.” The phenomenon is an anti-pattern that is unique to shops with fairly mature Dead Sea/Expert Beginner presences, and I’ll call it “Council of Elders.”

To understand what I mean by this, let’s consider what happens in an organization with a more vital software group full of engaged, competent, and improving engineers and not bogged down in the politics and mediocrity of seniority-based advancement. In such a group, natural divisions of labor emerge based on relative expertise. For example, you might have an effective team with the “database guy” and the “architecture gal” and “markup guru” and the “domain knowledge expert” or some such thing. Each resident expert handles decision making within his or her sphere of influence and trusts other team members to likewise do the right thing in their own areas of strength. There might be occasional debate about things that blur the lines or occur on the boundary between two experts, but that’s just part of a healthy group dynamic.

And truly, this is a healthy group. There is no dead weight, and all of the group members have autonomy within a subset of the work that is done, meaning that people are happy. And that happiness breeds mutual trust and productivity. I’m oversimplifying and prettying up the counterexample a bit, but it really is something of a virtuous cycle with team members happily playing to their own strengths and areas of interest.

TheCouncilBut what happens in particularly salty Dead Sea shops where authority comes neither from merit nor from expertise, but rather from tenure with the company? What happens when the roost is ruled by Expert Beginners? Well, for one thing, the lords and ladies here tend to guard their fiefdoms more jealously since expertise on the part of others is more threat than benefit. Perhaps more importantly and broadly, however, knowledge and expertise are devalued in favor of personal politics and influence with debates being won on the basis of “who are you and how loud/charming/angry/glib/etc. are you” rather than on the merit of ideas. The currency of Dead Sea departments is everything but ideas–in benevolent ones, it may be “how long have you been here” or “how nice a person are you,” and, in “high pressure culture” ones, it might simply be psychopathy or other cutthroat forms of “might makes right.” And with evaluation of ideas out the window, every council member is freed to hold forth as an expert on every topic, regardless of how much or little he knows about that topic. Nobody is going to dispute anything he says–non-members are cowed into submission and fellow members recognize the importance of putting on a unified public front since they want to be be able to do the same without being questioned.

If you put this political yeast in the oven and let it rise, some of the results are fairly predictable: idea stagnation, increasingly bone-headed solutions to problems, difficulty keeping talent, etc. But an interesting consequence isn’t necessarily intuitive–namely that you’ll wind up with a kind of cabal of long-tenured people that collectively make all decisions, however big or small. I call this the “Council of Elders,” and it’s like one of those Magic Eye paintings that’s hidden but couldn’t be more obvious once you see it.

The Council of Elders is sort of like the Supreme Court of the department, and it’s actually surprisingly democratic as opposed to the more expected ladder system which ranks people by years and months with the company (or if you’re a fan of The Simpsons, the system in the Stonecutters episode where all members of the club are assigned a numeric rank in the order of joining, which determines their authority). The reason that it’s democratic is that actually assigning rank based on years/months of tenure would unceremoniously poke a hole in any illusion of meritocracy. So the council generally makes entrance to the club a matter of tenure, but status within the club a shifting matter of alliances and status games once past the velvet rope.

The Council is fundamentally incapable of delegation or prioritization of decision making. Since entrance is simply a matter of “paying your dues” (i.e. waiting your turn) and largely earned by Expert Beginners, it’s really impossible to divide up decision making based on expertise. The members tend to be very good at (or at least used to) company politics and procedures but not much else. They mostly have the same ‘skill’ set. The lack of prioritization comes from the main currency in the group being status. If a decision, however small, is made by someone not on the Council, it threatens to undermine the very Council itself, so a policy of prevention is adopted and any attempts at circumvention are met with swift and terrible justice (in the form of whatever corporate demerits are in place).

Recognizing Your Elders

What does this council look like in the wild, and how do you know if one is pulling your strings? Here’s a set of symptoms that you’re being governed by a Council of Elders:

  • In any meeting convened to make a decision, the same group of people with minor variants is always present.
  • Members of the software group are frequently given vague instructions to “talk to so and so before you do anything because his cousin’s dog’s trainer once did something like that or something.”
  • There is a roster, emailed out or tacked up, of people who can “clear” you to do something (e.g. they give code reviews, approve time spent, etc.)
  • Sparks fly when something wasn’t “cleared” with someone, even when a few others “approved” it and it has apparently nothing to do with him. “Someone should have told me.” More than one or two people like this and you have a Council.
  • People in the group often feel caught between a rock and a hard place when involved in the political posturing of the Council (e.g. Yoda tells the intern Padiwan to implement the new hours tracking app using SQL Server, but then Mace Windu screams at him that this is a MySQL shop–the Council settles this matter later with no apologies)
  • There is a junta of team members that seem to do nothing but shuffle from meeting to meeting all day like mid-level managers.
  • There are regular meetings viewed by newbies and junior developers as rites of passage.

These are just easy ways to detect Councils in their natural habitat: “where there’s smoke, there’s fire” situations. But lest you think that I’m trying to paint every software department with senior developers that go to meetings with this brush, here is a list of specific criteria — “minimum Council Requirements,” if you will:

  • Lack of clearly defined roles on the group/team or else lack of clear definition for the assigned roles (e.g. there is no ‘architect’ or if there is, no one really knows what that means).
  • A “dues paying” culture and promotions, power and influence largely determined by length of tenure.
  • Lack of objective evaluation criteria of performance, skill and decision-making acumen (i.e. answers to all questions are considered subjective and matters of opinion).
  • Proposed changes from above are met with claims of technical infeasibility and proposed changes from juniors/newbies or other groups are met with vague refusals or simply aren’t acknowledged at all.
  • Actions of any significance are guarded with gatekeeper policies (official code reviews, document approval, sign-off on phases, etc).
  • Line manager tacitly approves of, endorses or is the product of the Council.
  • A moderate to large degree of institutional bureaucracy is in place.
  • New technologies, techniques, approaches, etc are met systematically with culturally entrenched derision and skepticism.

Not a Harmless Curiosity

At this point, you might think to yourself, “So what? It might not be ideal, but rewarding people with a little gratifying power in exchange for company loyalty is common and mostly harmless.” Or perhaps you’re thinking that I’m overly cynical and that the Council generally has good advice to dispense–wisdom won in the trenches. Of course all situations are different, but I would argue that a Council of Elders has a relatively noticeable and negative effect on morale, productivity, and general functioning of a group.

  • The Council is a legitimate bottleneck in its micromanagement of other team members, severely hampering their productivity even with the best assumption of its judiciousness
  • A group of people that spends all its time debating one another over how to rule on every matter doesn’t actually get much work done. The SCOTUS, for example, hasn’t represented a lot of clients lately, but that’s because their job is to be a ruling council–yours probably has titles like, “Senior Software Engineer.”
  • People lacking expertise but put in positions of authority tend to overcompensate by forming strong opinions and sticking to them stubbornly. A room full of people meeting this criterion is going to resemble the US House of Representatives with its gridlock more than a software team.
  • Major problems are likely to languish without solutions because the committee doesn’t do much prioritizing and is likely to be sidetracked for a few days by the contentious issue of what color to make the X for closing the window.
  • Important decisions are made based on interpersonal dynamics among the council rather than merit. Jones might have the best idea, but Jones shorted the check at lunch that day, so the other Council Members freeze him out.
  • Councils make it hard to give meaningful roles or titles to anyone and thus give rise to preposterous situations in which departments have eight ‘architects’ and five developers, or where a project manager decides what database to use while a DBA writes requirements. If everyone on the Council has authority and is an expert at everything, project roles are meaningless anyway.
  • Even under the best circumstances, democratic voting on software design trends toward mediocrity: see Design by Committee Anti-Pattern
  • People toiling away under the rule of a Council, if they don’t leave, will tend to wind up checked out and indifferent. Either that, or they’ll be assimilated into the Council culture.
  • The Council is a natural preservation agent of the Dead Sea problem. Being micromanaged by a team of people whose best qualification is being around for a while and negotiating power will separate the wheat from the chaff, but not in the good way.
  • The only thing that unites the Council is outside threats from above or below. If managers or newer members want change, the Council will lock together like a Praetorian Legion to preserve the status quo, however dysfunctional that may be.
  • Suggestions for improvement are construed with near universality as threats.

If you’re in a position to do so, I’d suggest breaking up the cartel. Figure out what people are good at and give them meaningful roles. Keep responsibilities divided, and not only will autonomy become more egalitarian, but also people tenured and new alike will develop complementary skills and areas of expertise. With people working together rather than forming shifting alliances, you’ll find prioritizing tasks, dividing up work, and getting things done becomes easier. The Council’s reptile brain will take over and it will fight you every step of the way when you try to disband it, but once you do, it will be good for everyone.

By

TDD: Simplest is not Stupidest

Where the Message Gets Lost In Teaching TDD

I recently answered a programmers’ stackexchange post about test-driven development. (As an aside, it will be cool when me linking to a SE question drives more traffic their way than them linking to me drives my way :) ). As I’m wont to do, I said a lot in the answer there, but I’d like to expand a facet of my answer into a blog post that hopefully clarifies an aspect of Test-Driven Development (TDD) for people–at least, for people who see the practice the way that I do.

One of the mainstays of showcasing test-driven development is to show some extremely bone-headed ways to get tests to pass. I do this myself when I’m showing people how to follow TDD, and the reason is to drive home the point “do the simplest thing.” For instance, I was recently putting together such a demo and started out with the following code:

This is how the code looked after going from the “red” to the “green” portion of the cycle. When I used CodeRush to define the IsEven method, it defaulted to throwing NotImplementedException, which constituted a failure. To make it pass, I just changed that to “return false.”

The reason that this is such a common way to explain TDD is that the practice is generally being introduced to people who are used to approaching problems monolithically, as described in this post I wrote a while back. For people used to solving problems this way, the question isn’t, “how do I get the right value for one,” but rather, “how do I solve it for all integers and how do I ensure that it runs in constant time and is the modulo operator as efficient as bit shifting and what do I do if the user wants to do it for decimal types should I truncate or round or throw an exception and whoah, I’m freaking out man!” There’s a tendency, often fired in the forge of undergrad CS programs, to believe that the entire algorithm has to be conceived, envisioned, and drawn up in its entirety before the first character of code is written.

So TDD is taught the way it is to provide contrast. I show people an example like this to say, “forget all that other stuff–all you need to do is get this one test passing for this one input and just assume that this will be the only input ever, go, now!” TDD is supposed to be fast, and it’s supposed to help you solve just one problem at a time. The fact that returning false won’t work for two isn’t your problem–it’s the problem of you forty-five seconds from now, so there’s no reason for you to bother with it. Live a little–procrastinate!

You refine your algorithm only as the inputs mandate it, and you pick your inputs so as to get the code doing what you want. For instance, after putting in the “return false” and getting the first test passing, it’s pretty apparent that this won’t work for the input “2”. So now you’ve got your next problem–you write the test for 2 and then you set about getting it to pass, say with “return target == 2″. That’s still not great. But it’s better, it was fast, and now your code solves two cases instead of just the one.

Running off the Rails

But there is a tendency I think, as demonstrated by Kristof’s question, for TDD teachers to give the wrong impression. If you were to write a test for 3, “return target == 2″ would pass and you might move on to 4. What do you do at 4? How about “return target == 2 || target == 4;”

So far we’ve been moving in a good direction, but if you take off your “simplest thing” hat for a moment and think about basic programming and math, you can see that we’re starting down a pretty ominous path. After throwing in a 6 and an 8 to the or clause, you might simply decide to use a loop to iterate through all even numbers up to int.MaxValue, or-ing a return value with itself to see if target is any of them.

Yikes! What went wrong? How did we wind up doing something so obtuse following the red-green-refactor principles of TDD? Two considerations, one reason: “simplest” isn’t “stupidest.”

Simplicity Reconsidered

The first consideration is that simple-complex is not measured on the same scale as stupid-clever. The two have a case-by-case, often interconnected relationship, but simple and stupid aren’t the same just as complex and clever aren’t the same. So the fact that something is the first thing you think of or the most brainless thing that you think of doesn’t mean that it’s the simplest thing you could think of. What’s the simplest way to get an empty boolean method to return false? “return false;” has no branches and one hardcoded piece of logic. What’s the simplest way that you could get a boolean method to return false for 1 and true for 2? “return target == 2″ accomplishes the task with a single conditional of incredibly simple math. How about false for 1 and true for 2 and 4? “return target % 2 == 0″ accomplishes the task with a single conditional of slightly more involved math. “return target == 2 || target == 4″ accomplishes the task with a single conditional containing two clauses (could also be two conditionals). Modulo arithmetic is more elegant/sophisticated, but it is also simpler.

Now, I fully understand the importance in TDD of proceeding methodically and solving problems in cadence. If you can’t think of the modulo solution, it’s perfectly valid to use the or condition and put in another data point such as testing for IsEven(6). Or perhaps you get all tests passing with the more obtuse solution and then spend the refactor phase refining the algorithm. Certainly nothing wrong with either approach, but at some point you have to make the jump from obtuse to simple, and the real “aha!” moment with TDD comes when you start to recognize the fundamental difference between the two, which is what I’ll call the second consideration.

The second consideration is that “simplest” advances an algorithm where “stupidest” does not. To understand what I mean, consider this table:

ConditionalClauseChart

In every case that you add a test, you’re adding complexity to the method. This is ultimately not sustainable. You’ll never wind up sticking code in production if you need to modify the algorithm every time a new input is sent your way. Well, I shouldn’t say never–the Brute Forces are busily cranking these out for you to enjoy on the Daily WTF. But you aren’t Brute Force–TDD isn’t his style. And because you’re not, you need to use either the green or refactor phase to do the simplest possible thing to advance your algorithm.

A great way to do this is to take stock after each cycle, before you write your next failing test and clarify to yourself how you’ve gotten closer to being done. After the green-refactor, you should be able to note a game-changing refinement. For instance:

MilestoneTddChart

Notice the difference here. In the first two entries, we make real progress. We go from no method to a method and then from a method with one hard-coded value to one that can make the distinction we want for a limited set of values. On the next line, our gains are purely superficial–we grow our limited set from distinguishing between 2 values to 3. That’s not good enough, so we can use the refactor cycle to go from our limited set to the complete set.

It might not always be possible to go from limited to complete like that, but you should get somewhere. Maybe you somehow handle all values under 100 or all positive or all negative values. Whatever the case may be, it should cover more ground and be more general. Because really, TDD at its core is a mechanism to help you start with concrete test cases and tease out an algorithm that becomes increasingly generalized.

So please remember that the discipline isn’t to do the stupidest or most obtuse thing that works. The discipline is to break a large problem into a series of comparably simple problems that you can solve in sequence without getting ahead of yourself. And this is achieved by simplicity and generalizing–not by brute force.

By

Wherefore Thou Shalt Fail at Software Requirements

On Being Deliberately Unclear

For whatever reason lately, I’ve been drawing a lot of inspiration from Programmers’ Stack Exchange, and today brings another such post. However, unlike other questions, reading this one seriously made me sad. The questions, the answers, its existence… all of it made me sad in the same kind of way that it’s vaguely depressing to think of how many hours of our lives we lose to traffic jams.

I had a conversation with a coworker once that involved a good-natured disagreement over the role of legalese in our lives. My (cynical) contention was that legalese has evolved in purpose–has even self-bastardized over the years–to go from something designed to clarify communication to something that is specifically used to hinder communication. To put it another way, by freezing the language of law in pedantic and flowery period-speak from the 18th century, lawyers have effectively created a throwback to the feudal practice of having a language for the nobles and a different language for the commoners. This language of the lawmaking nobles builds a bit of social job security in a way that writers of inscrutable code have picked up on–if you make it so no one understands your work and yet everyone depends on your work, then they have no choice but to keep you around. (Lawyers also have the benefit of being able to make laws to force you to need lawyers, which never hurts.)

My colleague appropriately pointed out that the nature of jurisprudence is such that it makes sense for lawyers to phrase things in documents in such a way that they can point to decided case law as precedent for what they’re doing. In other words, it’s ironically pragmatic to use language that no one understands because the alternative is that you write it in language less than 200 years old but then have to systematically re-defend every paragraph in various lawsuits later. I conceded this as a fair point, but thought it pointed to a more systemic flaw. It seems to me that any framework for bargaining between two parties that inherently makes communication harder is, ipso facto, something of a failure. He disagreed, being of the opinion that this old language is actually more precise and clear than today’s language. At this point, we agreed to disagree. I’ve since pondered the subject of language rules, language evolution, and precision. (As an aside, this bit of back-story is how the conversation went to the best of my recollection, so the details may be a little rough around the edges. It is not my intention to misrepresent anyone’s positions.)

Requirement Shock and Awe

Because of this pondering, my strange brain immediately linked the SE post with this conversation from some months back. I worked for a company once where the project manager thunderously plopped down some kind of 200-page Word document, in outline form, that contained probably 17,000 eerily and boringly similar statements:

  • 1.17.94.6b The system shall display an icon of color pink at such time as the button of color red button is clicked with the pointing device commonly known as a “mouse.”
  • 1.17.94.6c The system shall display an icon of color pink at such time as the button of color red is clicked with the pointing device commonly known as a “mouse,” 1st planet from Sol, Mercury, spins in retrograde, and the utilizer utilizes the menu option numbering 12.
  • 1.17.94.6d The system shall display an icon of color mauve at such time as the button of color purple is clicked with the pointing device commonly known as a “mouse.”
  • 1.17.94.6e The system shall require utilizers to change their underwear every half hour.
  • 1.17.94.6f The system shall require utilizers to wear underwear on the outside of their pants so that it can check.
  • 1.17.94.6g The system shall guarantee that all children under the age of 16 are now… 16.
  • 1.17.94.6h The system shalt not make graven images.

YeRequirements2A week later, when I asked a clarifying question about some point in a meeting, he furrowed his brow in disappointment and asked if I had read the functional-requirement-elaboration-dohickey-spec-200-pager, to which I replied that I had done my best, but, much like the dictionary, I had failed at complete memorization. I stated furthermore that expecting anyone to gain a good working understanding of the prospective software from this thing was silly since people don’t picture hypothetical software at that level of granularity. And besides, the document will be woefully wrong by the time the actual software is written anyway.

While I was at it with the unpopular opinions, I later told the project manager (who wasn’t the one that wrote this behemoth–just the messenger) that the purpose of this document seemed not to be clarity at all, but a mimicking of legalese. I gathered that the author was interested primarily in creating some kind of “heads I win, tails you lose” situation. This juggernaut of an artifact assured that everyone just kind of winged it with software development, but that when the barrel crashed at the bottom of the Waterfall, the document author could accuse the devs of not fulfilling their end of the bargain. Really, the document was so mind-numbing and confusing as to make understanding it a relative matter of opinion anyway. Having gotten that out in the open, I proceeded to write software for the release that was ahead of schedule and accepted, and I never looked at the intimidating requirements encyclopedia again. I doubt anyone ever did or will if they’re not looking for evidentiary support in some kind of blame game.

Well, time went by, and I filed that bit of silliness under “lol @ waterfall.” But there had been a coworker who was party to this conversation that mentioned something about the RFCs, or at least the notion of common definition, when I had been on my soapbox. That is, “shall” was used because it meant something specific, say, as compared to “will” or “might” or “should.” And because those specific meanings, apparently defined for eternity in the hallowed and dusty annals of the RFC, will live on beyond the English language or even the human race, it is those meanings we shall (should? might? must?) use. I mention these RFCs because the accepted answer to that dreadful Stack Exchange question mentioned them, clearing up once and for all and to all, including Bill Clinton, what the definition of “is” is (at least in the case of future tenses simple, perfect, continuous, and perfect continuous).

The only problem is that the RFCs weren’t the first place that someone had trotted out a bit of self-important window-dressing when it came to simple verb usage. Lawyers had beaten them to the punch:

First, lawyers regularly misuse it to mean something other than “has a duty to.” It has become so corrupted by misuse that it has no firm meaning. Second—and related to the first—it breeds litigation. There are 76 pages in “Words and Phrases” (a legal reference) that summarize hundreds of cases interpreting “shall.” Third, nobody uses “shall” in common speech. It’s one more example of unnecessary lawyer talk. Nobody says, “You shall finish the project in a week.” For all these reasons, “must” is a better choice, and the change has already started to take place. The new Federal Rules of Appellate Procedure, for instance, use “must,” not “shall.”

“Shall” isn’t plain English… But legal drafters use “shall” incessantly. They learn it by osmosis in law school, and the lesson is fortified in law practice.

Ask a drafter what “shall” means, and you’ll hear that it’s a mandatory word—opposed to the permissive “may”. Although this isn’t a lie, it’s a gross inaccuracy… Often, it’s true, “shall” is mandatory… Yet the word frequently bears other meanings—sometimes even masquerading as a synonym of “may”… In just about every jurisdiction, courts have held that “shall” can mean not just “must” and “may”, but also “will” and “is”. Increasingly, official drafting bodies are recognizing the problem… Many… drafters have adopted the “shall-less” style… You should do the same.

Bringing things back full circle, we have lawyers misusing a variant of “to be”–one of the most basic imaginable concepts in the English language–in order to sound more official and to confuse and intimidate readers. We have the world’s middle manager, pointy-haired types wanting to get in on that self-important action and adopting this language to stack the deck in CYA poker. And, mercifully, we have a group defining the new Federal Rules of Appellate Procedure who seems to value clarity over bombastic shows of rhetorical force designed to overpower objections and confuse skeptics. On the whole, we have a whole lot of people using language to impress, confuse, and intimidate, and a stalwart few trying to be precise.

A Failure of Both Clarity and Modernity

But believe it or not, the point of this post isn’t specifically to rag on people for substituting the odd “utilize” for “use,” sports-announcer-style, to try to sound smart. From academia to corporate meetings to dating sites (and courtrooms and requirements documents), that sort of thing is far too widespread for me to take it as a personal crusade. It might make me a hypocrite anyway to try. My point here is a more subtle one than that, and a more subtle one than pointing out the inherent, RFC futility of trying to get the world to agree to some made up definitions of various tense forms of “to be.”

The meat of the issue is that using slightly different variants of common words to emphasize things is a very dated way of defining requirements. Think of this kind of nonsense: “well… shall means that it’s the developer’s responsibility, but shalt means that it’s an act of God, and will means that project management has to help, and optional means optional and blah, blah blah…” What’s really going on here? I’d say what’s happening is some half-baked attempt to define concepts like “priority” and “responsible party” and “dependency.” You know, the kind of things that would do well, in, say, a database as fields. Pointy-haireds come for the semantic quibbling and they stay for the trumped-up buzz speak, so it winds up being harder than it should be to move away from a system where you get to bust out the fine verbal china and say impressive sounding things like “shall.”

The real problem here is that this silliness is a holdover from days when requirements were captured on Word Documents (or typewriters) as long, flowery prose and evaluated as a gestalt. But with modern ALM systems, work items, traceability tools, etc., this is colossal helping of fail. Imagine a requirement phrased as “when a user clicks the submit button, a JSON message is sent to the server.” That string can sit in a database along with a boolean that’s set to true when this accurately describes the world and false when it doesn’t. You want priority? How about an integer instead of bickering over whether it should be “JSON message {shall, will, ought to, might, mayhap, by-Jove-it-had-better, shalt, may, etc.} sent to the server”? When you’re coding, it’s an anti-pattern to parse giant chunks of text in a database field, so why isn’t it considered one to do it during requirements definition?

We have far better tools for capturing and managing requirements than word processors, pointless arguments over misappropriated grammar, standards used to define ARPANET in the 60’s, and polyester bell-bottoms. It’s 2013, so let’s not define and track requirements like it’s 1999… or 1969. Ditch the RFCs, the waterfall, the BS Bingo talk, and the whole nine yards if you want to get things done. But if you’d prefer just to focus on figuring out who to blame when you fail, then you should/shall/will/must/might/shalt not change a thing.

By

Productivity Add-Ins: Bruce Lee vs Batman

In the last few months, I’ve seen a number of tweets and posts decrying or at least cautioning against the use of productivity tools (e.g. CodeRush, ReSharper, and JustCode). The reasoning behind this is is generally some variant of the notion that such tools are more akin to addictive drugs than to sustainable life improvements. Sure, the productivity tool is initially a big help, but soon you’re useless without it. If, on the other hand, you had just stuck to the simple, honest, clean living of regular development, you might not have reached the dizzying highs of the drug, but neither would you have experienced crippling dependency and eventual rock bottom. The response from those who disagree is also common: insistence that use doesn’t equal dependence and the extolling of the virtues of such tools. Predictably, this back and forth usually degenerates into Apple v. Android or Coke v. Pepsi.

Before this degeneration, though, the debate has some fascinating overtones. I’d like to explore those overtones a bit to see if there’s some kind of grudging consensus to be reached on the subject (though I recognize that this is probably futile, given the intense cognitive bias of endowment effect on display when it comes to add-ins and enhancements that developers use). At the end of the day, I think both outlooks are born out of legitimate experience and motivation and offer lessons that can at least lead to deciding how much to automate your process with eyes wide open.

Also, in the interests of full disclosure, I am an enthusiastic CodeRush fan, so my own personal preference falls heavily on the side of productivity tools. And I have in the past plugged for it too, though mainly for the issue static analysis purpose rather than any code generation. That being said, I don’t actually care whether people use these tools or not, nor do I take any personal affront to someone reading that linked post, deciding that I’m full of crap, and continuing to develop sans add-ins.

The Case Against the Tools

There’s an interesting phenomenon that I’ve encountered a number of times in a variety of incarnations. In shops where there is development of one core product for protracted periods of time, you meet workaday devs who have not actually clicked “File->New” (or whatever) and created a new project in months or years. Every morning they come in at 9, every evening they punch out at 5, and they know how to write code, compile it, and run the application, all with plenty of support from a heavyweight IDE like Eclipse or Visual Studio. The phenomenon that I’ve encountered in such situations is that occasionally something breaks or doesn’t work properly, and I’ve said, “oh, well, just compile it manually from the command line,” or, “just kick off the build outside of the IDE.” This elicits a blank stare that communicates quite effectively, “dude, wat–that’s not how it works.”

When I’ve encountered this, I find that I have to then have an awkward conversation with a non-entry-level developer where I explain the OS command line and basic program execution; the fact that the IDE and the compiler are not, in fact, the same thing; and other things that I would have thought were pretty fundamental. So what has gone wrong here? I’d say that the underlying problem is a classic one in this line of work–automation prior to understanding.

AngryArchLet’s say that I work in a truly waterfall shop, and I get tired of having to manually destroy all code when any requirement changes so that I can start over. Watching for Word document revisions to change and then manually deleting the project is a hassle that I’ve lived with for one day too long, so I fire up the plugin project template for my IDE and write something that runs every time I start. This plugin simply checks the requirements document to see if it has been changed and, if so, it deletes the project I’m working on from source control and automatically creates a new, blank one.

Let’s then say this plugin is so successful that I slap it into everyone’s IDE, including new hires. And, as time goes by, some of those new hires drift to other departments and groups, not all of which are quite as pure as we are in their waterfall approach. It isn’t long before some angry architect somewhere storms over, demanding to know why the new guy took it upon himself to delete the entire source control structure and is flabbergasted to hear, “oh, that wasn’t me–that’s just how the IDE works.”

Another very real issue that something like a productivity tool, used unwisely, can create is to facilitate greatly enhanced efficiency at generating terrible code (see “Romance Author“). A common development anti-pattern (in my opinion) that makes me wince is when I see someone say, “I’m sure generating a lot of repetitive code–I should write some code that generates this code en masse.” (This may be reasonable to do in some cases, but often it’s better to revisit the design.) Productivity tools make this much easier and thus more tempting to do.

The lesson here is that automation can lead to lack of understanding and to real problems when the person benefiting doesn’t understand how the automation works or if and why it’s better. This lack of understanding leads to a narrower view of possible approaches. I think a point of agreement between proponents and opponents of tools might be that it’s better to have felt a pain point before adopting the cure for it rather than just gulping down pain medication ‘preventatively’ and lending credence to those saying the add-ins are negatively habit-forming. You shouldn’t download and use some productivity add-in because all of the cool kids are doing it and you don’t want to be left out of the conversations with hashtag #coderush

The Case for the Tools

The argument from the last section takes at face value the genuine concerns of those making it and lends them benefit of the doubt that their issue with productivity tools is truly concern for bad or voodoo automation. And I think that requires a genuine leap of faith. When it comes to add-ins, I’ve noticed a common thread between opponents of that and opponents of unit tests/TDD–often the most vocal and angry opponents are ones that have never tried it. This being the case, the waters become a little bit muddied since we don’t know from case to case if the opponent has consistently eschewed them because he really believes his arguments against them or if he argues against them to justify retroactively not having learned to use them.

And that’s really not a trivial quibble. I can count plenty of detractors that have never used the tools, but what I can’t recall is a single instance of someone saying, “man, I used CodeRush for years and it really made me worse at my job before I kicked the habit.” I can recall (because I’ve said) that it makes it annoying for me to use less sophisticated environments and tooling, but I’d rather the tide rise and lift all of the boats than advocate that everybody use notepad or VI so that we don’t experience feature envy if we switch to something else.

NorthPolePigeonThe attitude that results from “my avoidance of these tools makes me stronger” is the main thing I was referring to earlier in the post when I mentioned “fascinating overtones.” It sets the stage for tools opponents to project a mix of rugged survivalist and Protestant Work Ethic. Metaphorically speaking, the VI users of the world sleep on a bed of brambles because things like beds and not being stabbed while you sleep are for weaklings. Pain is gain. You get the sense that these guys refuse to eat anything that they didn’t either grow themselves or shoot using a homemade bow and arrow fashioned out of something that they grew themselves.

But when it comes to programming (and, more broadly, life, but I’ll leave that for a post in a philosophy blog that I will never start) this affect is prone to reductio ad absurdum. If you win by leaving productivity tools out of your Visual Studio, doesn’t the guy who uses Notepad++ over Visual Studio trump you since he doesn’t use Intellisense? And doesn’t the person who uses plain old Notepad trump him, since he doesn’t come to rely on such decadent conveniences as syntax highlighting and auto-indentation? And isn’t that guy a noob next to the guy who codes without a monitor the way that really, really smart people in movies play chess without looking at the board? And don’t they all pale in comparison to someone who lives in a hut at the North Pole and sends his hand-written assembly code via carrier pigeon to someone who types it into a computer and executes it (he’s so hardcore that he clearly doesn’t need the feedback of running his code)? I mean, isn’t that necessary if you really want to be a minimalist, 10th degree black-belt, Zen Master programmer–to be so productive that you fold reality back on itself and actually produce nothing?

The lesson here is that pain may be gain when it comes to self-growth and status, but it really isn’t when the pain makes you slower and people are paying for your time. Avoiding shortcuts and efficiency so that you can confidently talk about your self reliance not only fails as a value-add, but it’s inherently doomed to failure since there’s always going to be some guy that can come along and trump you in that “disarms race.” Doing without has no intrinsic value unless you can demonstrate that you’re actively being hampered by a tool.

So What’s the Verdict?

I don’t know that I’ve covered any ground-breaking territory except to point out that both sides of this argument have solutions to real but different problems. The minimalists are solving the problem of specious application of rote procedures and lack of self-reliance while the add-in people are solving the problem of automating tedious or repetitive processes. Ironically, both groups have solutions for problems that are fundamental to the programmer condition (avoiding doing things without knowing why and avoiding doing things that could be done better by machines, respectively). It’s just a question of which problem is being solved when and why it’s being solved, and that’s going to be a matter of discretion.

Add-in people, be careful that you don’t become extremely proficient at coding up anti-patterns and doing things that you don’t understand. Minimalist people, recognize that tools that others use and you don’t aren’t necessarily crutches for them. And, above all, have enough respect for one another to realize that what works for some may not work for others. If someone isn’t interested in productivity tools or add-ins and feels more comfortable with a minimalist setup, who are any of us to judge? I’ve been using CodeRush for years, and I would request the same consideration–please don’t assume that I use it as a template for 5,000 line singletons and other means of mindlessly generating crap.

At the end of the day, whether you choose to fight bad guys using only your fists, your feet, and a pair of cut-off sweat shorts or whether you have some crazy suit with all manner of gizmos and gadgets, the only important consideration when all is said and done is the results. You can certainly leave an unconscious and dazed pile of ne’er-do-wells in your wake either way. Metaphorically speaking, that is–it’s probably actually soda cans and pretzel crumbs.

Acknowledgements | Contact | About | Social Media