Without Precedence

Introducing Overflow.net

by Morten Christiansen on 28-09-2015 at 19:20 | comments [0] | posted in Open Source

This post introduces my new (and first) proper open source .NET library, Overflow.net.

It all started last December when I was working on a service for transferring salary files using a daily scheduled job. These files are used for paying out the salary and pensions for the employees of a number of our customers. It goes without saying that such a task must never fail -- and when it does, it must be very easy to figure out what went wrong. Preferably a fix will be released before people begin wondering where their money is.

Several times during my career I have been in a position where I needed to write code for executing these complex multi-step workflows complete with logging, error handling and work-resuming capabilities. You often end up hacking something together that works by the power of prayers and luck but in this case that was just not good enough. My solution was to build a small framework for executing a series of steps in a hierarchical workflow. The infrastructure would handle all the heavy lifting of logging, error handling, etc. allowing me to focus on the business logic instead.

Most of the time you don't have the luxury of building an advanced infrastructure when you can get away with much less. In fact, despite the severity of my requirements, there where many things about my solution that nagged me. Then it was that I figured that it made sense to build the infrastructure as a stand alone project and make it open source. This way, I would have a great tool handy when next I came into this kind of situation and others could benefit from it as well.

It should be mentioned that there exist a number of different libraries out there for describing workflows. They all have different takes on the problem but none of them met my goals. They where quite simple: it must have a minimalistic declarative syntax, allowing me to focus on the business logic with as little distraction as possible and it must be configured and executed entirely in code. It must also allow me to inspect an execution in detail as it usually deals with unattended jobs.

The library centers around the concept of the operation, a hierarchical unit of work that can have any number of child operations. The root operation will be called the workflow, though there is nothing special about it. The amount of work captured in an operation can be as little or as much as desired, but should represent a single conceptual piece of work. If the operation is sufficiently complex, you might want to split it into discreet steps or sub operations. The operation must implement this interface:

  1.     public interface IOperation
  2.     {
  3.         IEnumerable<ExecutionInfo> ExecutedChildOperations { get; }
  5.         void Initialize(WorkflowConfiguration configuration);
  6.         IEnumerable<IOperation> GetChildOperations();
  7.         void Execute();
  8.     }

In practice your operations will extend the `Operation` class which implements the interface and allows you to hook into its functionality by overriding a number of methods. It also provides useful functionality for common tasks such as instantiating child operations. I think the best way to get a feel for it is just to show some code, so here is a more or less complete (if quite useless) example.

  1.     class ImportUserWorkflow : Operation
  2.     {
  3.         public override IEnumerable<IOperation> GetChildOperations()
  4.         {
  5.             yield return Create<LoadUserDataFromExternalDatabaseOperation>();
  6.             foreach (var externalUser in GetChildOutputValues<ExternalUser>())
  7.                 yield return Create<ImportExternalUserOperation, ExternalUser>(externalUser);
  8.         }
  9.     }
  10.     [Retry]
  11.     class LoadUserDataFromExternalDatabaseOperation : Operation, IOutputOperation<List<ExternalUser>>
  12.     {
  13.         private readonly IExternalDatabase _db;
  14.         private Action<List<ExternalUser>> _onReceiveExternalUsers;
  15.         public LoadUserDataFromExternalDatabaseOperation(IExternalDatabase db)
  16.         {
  17.             _db = db;
  18.         }
  19.         public void Output(Action<List<ExternalUser>> onReceiveExternalUsers)
  20.         {
  21.             _onReceiveExternalUsers = onReceiveExternalUsers;
  22.         }
  23.         protected override void OnExecute()
  24.         {
  25.             var newUsersToImport = _db.Users.Where(u => !u.IsImported).ToList();
  26.             _onReceiveExternalUsers(newUsersToImport);
  27.         }
  28.     }
  29.     [ContinueOnFailure, Atomic]
  30.     class ImportExternalUserOperation : Operation, IInputOperation<ExternalUser>
  31.     {
  32.         private ExternalUser _externalUser;
  33.         public void Input(ExternalUser externalUser)
  34.         {
  35.             _externalUser = externalUser;
  36.         }
  37.         public override IEnumerable<IOperation> GetChildOperations()
  38.         {
  39.             yield return Create<SanitizeExternalUserOperation, ExternalUser>(_externalUser);
  40.             yield return Create<PersistImportedUserOperation>();
  41.             yield return Create<MarkUserAsImportedInExternalDatabaseOperation, ExternalUser>(_externalUser);
  42.             yield return Create<SendWelcomeEmailOperation>();
  43.         }
  44.     }
  45.     class SanitizeExternalUserOperation : Operation, IInputOperation<ExternalUser>, IOutputOperation<User>
  46.     {
  47.         private ExternalUser _externalUser;
  48.         private Action<User> _onReceiveUser;
  49.         public void Input(ExternalUser externalUser)
  50.         {
  51.             _externalUser = externalUser;
  52.         }
  53.         public void Output(Action<User> onReceiveUser)
  54.         {
  55.             _onReceiveUser = onReceiveUser;
  56.         }
  57.         protected override void OnExecute()
  58.         {
  59.             _onReceiveUser(Sanitize(_externalUser));
  60.         }
  61.         private User Sanitize(ExternalUser externalUser)
  62.         {
  63.             // Do some actual work here
  64.             return new User();
  65.         }
  66.     }
  67.     [Retry]
  68.     class PersistImportedUserOperation : Operation, IInputOperation<User>
  69.     {
  70.         private readonly IDatabase _db;
  71.         private User _user;
  72.         public PersistImportedUserOperation(IDatabase db)
  73.         {
  74.             _db = db;
  75.         }
  76.         public void Input(User user)
  77.         {
  78.             _user = user;
  79.         }
  80.         protected override void OnExecute()
  81.         {
  82.             _db.Save(_user);
  83.         }
  84.     }
  85.     [Retry]
  86.     class MarkUserAsImportedInExternalDatabaseOperation : Operation, IInputOperation<ExternalUser>
  87.     {
  88.         private readonly IExternalDatabase _db;
  89.         private ExternalUser _externalUser;
  90.         public MarkUserAsImportedInExternalDatabaseOperation(IExternalDatabase db)
  91.         {
  92.             _db = db;
  93.         }
  94.         public void Input(ExternalUser externalUser)
  95.         {
  96.         _externalUser = externalUser;
  97.         }
  98.         protected override void OnExecute()
  99.         {
  100.             _externalUser.IsImported = true;
  101.             _db.Save(_externalUser);
  102.         }
  103.     }
  104.     [CompensatingOperation(typeof(SendWelcomeSmsOperation))]
  105.     class SendWelcomeEmailOperation : Operation, IInputOperation<User>
  106.     {
  107.         private readonly IEmailService _emailService;
  108.         private User _user;
  109.         public SendWelcomeEmailOperation(IEmailService emailService)
  110.         {
  111.             _emailService = emailService;
  112.         }
  113.         public void Input(User user)
  114.         {
  115.             _user = user;
  116.         }
  117.         protected override void OnExecute()
  118.         {
  119.             _emailService.SendWelcomeEmail(_user);
  120.         }
  121.     }
  122.     class SendWelcomeSmsOperation : Operation, IInputOperation<User>
  123.     {
  124.         private readonly ISmsService _smsService;
  125.         private User _user;
  126.         public SendWelcomeSmsOperation(ISmsService smsService)
  127.         {
  128.             _smsService = smsService;
  129.         }
  130.         public void Input(User user)
  131.         {
  132.             _user = user;
  133.         }
  134.         protected override void OnExecute()
  135.         {
  136.             _smsService.SendWelcomeMessage(_user);
  137.         }
  138.     }

This should illustrate the idea of how you compose a set of operations into a workflow. At this point you might be wondering why we had to jump through all those hoops simply to instantiate an object with a few properties and send it to a method on some service. You might think that we have just added complexity without adding value. However, it is not in the happy path that the strength of the library lies but rather the way it supports easily handling different error scenarios. Notice the attributes added to most of the operation classes. Hopefully, they provide a good idea about what they do. For example, the `Atomic` attribute wraps the operation in a transaction that is committed if the operation completes without throwing any exceptions. Another attribute is `Retry` which by default retries an operation 3 times with a second delay in case of any exception. These attributes know which order it makes sense to execute in, so the retry logic executes before the transaction gets rolled back, giving a final chance to keep everything going.

Writing ad hoc code to have the same functionality as the above code would be an unreadable try/catch mess where the execution flow would be very hard to follow. You might be able to write it in a decent fashion but I would submit that it is not possible without spending a fair bit of time on some infrastructure code to serve some of the same purposes.

In addition to the capabilities for taking corrective action at runtime, the logging facilities allow you to easily get an overview of an execution and what went wrong. Consider the following log based on the above code, where a single external user is imported, and the email service failed to send the notification.

  1.     ImportUserWorkflow
  2.     {
  3.         LoadUserDataFromExternalDatabaseOperation [duration: 77ms]
  4.         ImportExternalUserOperation
  5.         {
  6.             SanitizeExternalUserOperation [duration: 0ms]
  7.             PersistImportedUserOperation [duration: 0ms]
  8.             MarkUserAsImportedInExternalDatabaseOperation [duration: 0ms]
  9.             SendWelcomeEmailOperation
  10.             {
  11.                 Error [Exception]: Could not connect to smtp server.
  12.                 CompensatingOperationBehavior: Executing compensating operation
  13.                 SendWelcomeSmsOperation [duration: 0ms]
  14.                 ContinueOnFailureBehavior: Error swallowed
  15.             } [duration: 1.840ms]
  16.         } [duration: 1.908ms]
  17.     } [duration: 1.997ms]

This was a brief introduction to the Overflow.net library; I hope you find it useful. Head over to the project Github page to read the docs and get the code or go view the Trello board with upcoming features.

The future is here and it is functional

by Morten Christiansen on 04-09-2013 at 18:44 | comments [0] | posted in Learning, Programming Languages

In recent months I had been getting this increasing sense that I was missing out on something special. From various online channels I was hearing about the greatness of C#’s lesser know cousin F#. F# is another .NET language but instead of being an object oriented language it has aspects from both the object oriented and functional styles. While most of the developers I talk to about F# shrug it off as something strange or just plain don’t know about it, I have been fascinated by it for several years, though I never got around to do anything about it.

What really piqued my interest was this short, but thought provoking blog post where the author points out why the functional programming style, and F# specifically has helped him eliminate most of his bugs. A little while later, I was listening to the excellent .NET Rocks podcast, where they had a guy named Kit Eason on the show. He talked about how he had gone over to F# for his professional work and how well it had worked at his company. While there were many positive things said about F# in the show, I think what got to me was the short time he needed to master the new paradigm. Kit mentioned that he was more or less up to full speed after two weeks of training, and according to himself he was no extraordinary developer.

Learning F#

Thusly motivated, I began looking for resources on how to learn the language and quickly stumbled upon the Try F# website by Microsoft. This is a great resource for getting a feel for the syntax and how the language behaves. It walks you through some of the basic concepts with small code samples that you can experiment with in an interactive F# console directly in the browser. When you get past the most basic examples, it quickly becomes obvious that you have to rotate your brain a bit to get to grips with the concepts and alien syntax of the F# language. All I can say is to have patience, it definitely gets better :)

At this point I had an understanding of the language syntax but no feeling for how to solve actual problems in it. To remedy this I turned to good old Project Euler, a set of mathematical problems designed to be solved in some programming language. I solved about the first ten problems and to make sure my solutions weren’t too awful, I found a website with solutions to the problems in F# where I could cross reference them. At this point, the outline of a greater understanding had begun coalescing in my mind though I was still ignorant of how to use the language for anything but isolated tasks.

Next, my exploration took me to an introduction to the benefits of F# from the point of view of a C# developer, in a whopping 30 step guide. This website, F# For Fun and Profit, is without a doubt the best resource I have found on the web for learning F#. Browsing the site, you will find many articles on more high level topics such as the excellent 8-part Designing With Types series. Not only is the technical content of the site very good, but the author excels at communicating new concepts and there was very little that I did not get a good understanding of after a single read through – and this is definitely not a given for F#.

Having spent a few days going through these resources I was done with the theory and ready to get my hands dirty with some real code. Instead of just beginning on something new I began porting code from several of my C# projects into F#. This has provided a great opportunity to compare the two styles of programming and I have been happy to observe that my experiences matched many of the positive things I had heard about the language. You can check out my first conversion project where I took my old Resizer project and converted it to a version with both an F# and C# implementation.

So do I like it?

Yes! I still have to think rather hard about writing idiomatic F# code sometimes, but it certainly is a promising language where many problems tend to have more elegant and succinct solutions compared to C#. While it has certain drawbacks such as poor support for the UI tools in Visual Studio, you can always limit yourself to writing some of your assemblies in F# – after all it is all just .NET code.

One of the great strengths of the language is for building your domain models. The type system (and type inference) is much more powerful than in C# and many aspects of your models can be directly encoded in the type system, making them very expressive and preventing them from getting into inconsistent states. I won’t go into much more detail about why the language is great right now, but a little thing that I absolutely love about the language is how string literals allow you to write super readable test names.


Note: The natural format of the assertions on the last two lines is facilitated by the library FsUnit.

Visual Studio Tip

One thing that might throw you off at first is that Visual Studio has very basic F# solution support. Due to how F# projects are compiled, the files must appear in the order in which they are referenced, in one long flat list. At first I was horrified at the thought of managing large projects in this way but fortunately a little hacking in the fsproj file can fix the worst of the issues. All it takes is for you to change the name of a source file to include the desired folder (which must exist on disk), and it works as you are used to. At this point you can add new files to the folder from the Solution Explorer but have to create new root folders in the same manner. This makes it bearable but it certainly is not the most polished experience. In case you’re using the Developer Preview version of Visual Studio 2013 as I am, do me a favor of letting Microsoft know how you feel about this using their feedback feature at the top right corner.


How to mark required fields with jQuery Validate

by Morten Christiansen on 16-02-2012 at 23:44 | comments [2] | posted in JavaScript

Today I was tasked with marking all the required field in an ASP.NET MVC application with asterisks, next to the relevant labels. Initially it seemed that there would be no other solution than to go through every view and add it by hand. Fortunately, I realized that there was a far better solution, one that would only require 13 lines of code. We use the jQuery Validation plugin for client-side validation and this is configured with a number of validation rules. I realized that if I could hook into these rules, I had all the information that I needed.

The concept behind this solution is simple, really. You just need to intercept the initializing logic that sets up the validation rules and update the labels as needed. The code in its entirety can be seen below.

  1. var validateMethod = $.fn.validate;
  2. $.fn.validate = function (o) {
  3.     if (o && o.rules) {
  4.         for (var name in o.rules) {
  5.             var rule = o.rules[name];
  6.             if (rule.required === true) {
  7.                 var label = $('label[for=' + name + ']');
  8.                 label.text(label.text() + "*");
  9.             }
  10.         }
  11.     }
  12.     return $.proxy(validateMethod, this)(o);
  13. };

The code assigns the original validation method to a variable, saving it for later, while we assign the plugin a new method. This method contains our logic, ending up executing the original method. Notice that we must use the jQuery proxy function for assigning the target element to the this property inside the validation function as it expects. That's all there is to it. Of course, you can check for other types of validation rules and annotate the form fields accordingly if you need to. Just execute this code at any time before initializing your validation rules.

Refactoring: Named boolean arguments

by Morten Christiansen on 22-01-2012 at 12:29 | comments [0] | posted in Best Practices, Refactoring

This is another article in my series on refactoring code to make it more readable and maintainable

Boolean arguments have a nasty tendency of making method calls very hard to read, especially when several of them are required. For example, consider this piece of code I just ran across:

  1. new SchemaExport(Configuration).Execute(true, true, false, true, session.Connection, Console.Out);

This might not by a typical example, but it clearly illustrates the problem. Just looking at it, you don't really know much about what this line of code does. Of course, your IDE can probably help you figure it out quickly enough, but several such lines can it makes it very hard to quickly figure out a section of code. I've run across variations of this issue countless times and have come to use one of two approaches to avoid this, depending on the circumstances.

The first is to use named variables, a feature of C#. Alternatively, you can declare a variable and supply it to the method. This makes for much clearer code:

  1. // named arguments
  2. var users = service.GetUsers(includeTemporary: true, loadDocuments: true);
  4. // named variables as arguments
  5. var includeTemporary = true;
  6. var loadDocuments = true;
  7. var users = service.GetUsers(includeTemporary, loadDocuments);

Most of the time, I find this to be a suitable solution. Sometimes, however, the boolean argument has such an impact on the implementation of the method that we're better served splitting it in two. This will keep each of the methods focused on a single responsibility making it easier to change one part without affecting the other. In the example below, the method is split into two because the notions of a document and a draft are, while related, not treated the same at all. Therefore, a more appropriate API would have methods for getting either type separately.

  1. // wrong way (because drafts and normal documents are never treated as the same thing)
  2. var documents = service.GetDocuments(includeDrafts: false);
  3. var documentsAndDrafts = service.GetDocuments(includeDrafts: true);
  5. // better way
  6. var documents = service.GetDocuments();
  7. var drafts = service.GetDrafts();

If needed, the implementations of the two methods can rely on secondary methods for any shared logic.

The IDE is dead; Long live the IDE

by Morten Christiansen on 31-10-2011 at 22:55 | comments [0] | posted in Programming

A while ago, I stumbled upon the ASP.NET Uservoice page for gathering feature requests and suggestions. Among the most popular requests was something odd, a feature named Code Bubbles. This did not sound like anything to do with web programming, but its great popularity sparked my interest. Fortunately, the suggestion featured a link to a website explaining the concept, including a demonstration video… and minds were blown!

These guys have completely reimagined the IDE, laying down the shackles of technology that have held back current efforts. Sounds dramatic? That's because it is. Abstractions have been added to allow more freeform ways to organize code without being limited by the notion of files and the project structure as we know it. The central theme of the work is to have all the code you need on your screen, and nothing else, working down to the method level – the groupings of code that give the concept its name. A host of features have been developed for making the workflow of navigating to related code effortless. But alas, words prove too poor a medium as I try to explain this wonder. Go, spend the next 8 minutes of your life watching the demonstration video of a working Java IDE exploring the Code Bubble idea. Here are their own words, describing the concept, but do follow the link to see the video and read more:

New life emergesDevelopers spend significant time reading and navigating code fragments spread across multiple locations. The file-based nature of contemporary IDEs makes it prohibitively difficult to create and maintain a simultaneous view of such fragments. We propose a novel user interface metaphor for code understanding and maintanence based on collections of lightweight, editable fragments called bubbles, which form concurrently visible working sets.

The essential goal of this project is to make it easier for developers to see many fragments of code (or other information) at once without having to navigate back and forth. Each of these fragments is shown in a bubble.

A bubble is a fully editable and interactive view of a fragment such as a method or collection of member variables. Bubbles, in contrast to windows, have minimal border decoration, avoid clipping their contents by using automatic code reflow and elision, and do not overlap but instead push each other out of the way. Bubbles exist in a large, pannable 2-D virtual space where a cluster of bubbles comprises a concurrently visible working set. Bubbles support a lightweight grouping mechanism, and further support connections between them.

A quantitative user study indicates that Code Bubbles increased performance significantly for two controlled code understanding tasks. A qualitative user study with 23 professional developers indicates substantial interest and enthusiasm for the approach, despite the radical departure from what developers are used to.

I'm absolutely in love with the idea that you have a sort of working space with all the methods and structures you are working on, allowing you to shift between them as you work on different tasks. Some urgent bug just came up? No problem, just move to a new work space and begin stepping through the affected code, building up the new working space.

Working on large projects, I'm sometimes tempted to bundle too much functionality into a single class, just to have all the related code at hand. As a project grows, the number of files that are relevant to some piece of functionality can grow very high, especially when you have to include all the layers of the application. What Code Bubbles does that looks so promising, is to put all the related code in a single screen or at least scrollable area. The calling hierarchy is clearly defined and it becomes easy to follow the thread of execution as it goes from class to class and module to module.

After finding this, I've begun noticing activity regarding the project across the web. One such this is the very interesting Debugging Canvas project from Microsoft Research. It is actually the result of a collaboration between Microsoft and the people behind Code Bubbles in an effort to bring the new ideas to Visual Studio. It's a limited part of the whole Code Bubbles concept but I'm thrilled to see it taken seriously by Microsoft. After all, until it lives in Visual Studio, it's largely irrelevant to me in practice. There is also a video for the Debugging Canvas on their site, so take a look at that as well. You can go install it now as an extension to Visual Studio, that is, if you have the Ultimate version of Studio.

I, for one, welcome our new Code Bubble overlords.