Matt Jackson

So I’ve been messing around building a Dependency Injection container in my spare time. Mainly for my own amusement, but also to practice designing DSLs and to play with some C# 4 features like anonymous objects. 

One of the first problems I came across was with requesting an interface that had been registered with a more specific generic type.

We have a representation of the registration of interface and its implementation via a factory method shown below:

   public class FactoryRegistration : ObjectRegistration
    {
        public FactoryRegistration<T>(Type forType, Func<T> factoryFunction) : base(forType)
        {
            FactoryFunction = factoryFunction;
        }

        public Func<T> FactoryFunction { get; private set; } 
    }

      public T Resolve<T>()
        {
            ...snip...

            var factoryRegistration = registration as FactoryRegistration<T>;
            return (T) factoryRegistration.FactoryFunction();
        }

   new FactoryRegistration<Class>(typeof(IClass), () => new Class());

  resolver.Resolve<IClass>();

  var factoryRegistration = registration as FactoryRegistration<IClass>;
  return (T) factoryRegistration.FactoryFunction();

   public Giraffe MakeGiraffe() {
           return new Giraffe();
   }

   Func<Animal> someFunc = MakeGiraffe();

    public interface IFactoryRegistration<out T>
    {
        Func<T> FactoryFunction { get; }
    }

    public class FactoryRegistration : ObjectRegistration, IFactoryRegistration<T>
    {
        public FactoryRegistration<T>(Type forType, Func factoryFunction) : base(forType)
        {
            FactoryFunction = factoryFunction;
        }

        public Func<T> FactoryFunction { get; private set; } 
    }

      public T Resolve<T>()
        {
            ...snip...

            var factoryRegistration = registration as IFactoryRegistration<T>;
            return (T) factoryRegistration.FactoryFunction();
        }

Those cheeky little devils at Swisscom have found a novel way of restricting wifi sharing of their 3G dongles, resulting in the following error message when attempting to connect to a network in Windows 7:

“Your network administrator has blocked you from connecting to this network”

With a little searching I came across this solution, which allows you to fix it via an Administrators Command Prompt as follows:

netsh wlan delete filter permission=denyall networktype=infrastructure 
netsh wlan delete filter permission=denyall networktype=adhoc

Enjoy

(continued from Part 2)

Test 2

This was centred around the choice of disk drive. We’re building a slightly different project from before, but similar size (about 100k LOC) and structure (about 55 projects). This main difference this time was that MsBuild was being executed with the following property:

OutDir=c:\A\Single\Output\Directory

So we are cutting out a LOT of I/O. Every assembly, including the lib ones, are being copied only once per run. So this negates a lot of the purported benefits of SSD and RAM drives, but IMO it ends up being much more indicative of the kind of improvements you can get from good architecture (refer to Test 1). Finally, we are also cleaning each bin & obj directories before each pass:

HDD - 5 builds in 3:00 mins @ 36 seconds each

SSD - 5 builds in 2:37 mins @ 31 seconds each (~15% faster)

RAM* - 5 builds in 2:20 mins @ 28 seconds each (~23% faster)

I also ran another RAM disk run using the original project structure in Test 1

10 builds in 03:06 @ 00:18 per build (~80% faster than the original 91 seconds)

* The RAM Disk used can be found here: http://www.ltr-data.se/opencode.html/#ImDisk

Conclusions

So what can we infer from these results? Well, quite a lot really - but as always they come with caveats.

1. I/O is a killer. Getting rid of all those copy local references took nearly 90% off the build time.

2. Csc.exe is fast. Damn fast. Spinning up all those Win32 subprocesses costs a lot of time, and when you only have to do it once, the compile times drop to near 0.

3. Faster drives help. But they only Band-Aid over the symptoms, you’ll get much better results by addressing the root cause.

Now obviously I’m not suggesting that you go and refactor your entire codebase into a single project file. There are lots of things to consider when structuring .NET solutions; the make-up of your deployable artefacts, separating test & production code, vertical & horizontal partitioning of functionality, “code re-use” (although the alleged benefits of this is a topic for another day) etc etc. However, its worth keeping in mind, especially early in the project, the potential consequences of design choices. 

Next time, I’ll be looking at the impact that CI tools can have on the speed of your build process.

PS. I discovered another little gem today. Sick of all those XML IntelliSense files clogging up your output folders? Try this little nugget, and shave 40% off your artefact sizes:

MsBuild.exe YourBuild.file /p:GenerateDocumentation=false;AllowedReferenceRelatedFileExtensions=none

(Continued from Part 1)

So, nothing like some hard figures to help sharpen the mind…

Test 1

This test was centred around project structure… that is, the number of csproj’s and the type of references each held to each other.

Our test machine was an i7 with 8 gigs of RAM and a 7200pm HDD, each run was held after a fresh reboot, no other programs running (save a terminal session). MsBuild was executed from within a small python script, and the timing information was output to the console. The measurements which follows aren’t particularly scientific, however I hope they serve as an indication of what is possible.

Original Structure (our baseline)

This was a solution file of approximately ~80 projects & 125k LOC. There was quite a number of projects with only a handful of .cs files, and every project file was set to CopyLocal = true for its non-mscorlib references. This basically amounted to a collective output of thousands of assemblies, with many copies of the same third-party DLL’s in each output directory (think, 100’s of copies of nhibernate.dll & nservicebus.dll). Our build times were:

10 builds in 15:18 mins @ 1:31 mins per build

Assembly/Weak references (CopyLocal = false)

That CopyLocal setting (or <private> node in the csproj XML) that I mentioned above was resulting in a huge amount of file I/O for each and every build. So, following Patrick Smacchia’s advice from the article I linked to in the first part of this series, I converted every reference (via a little tool I wrote) to CopyLocal = false, and ran another test run. An 81% improvement… not bad.

10 builds in 3:08 mins @ 0:18 mins per build

Single Project file

Watching all those MsBuild instances scroll up my terminal window (apart from being ridiculously hypnotic), I noticed something else. MsBuild shelling out to csc.exe takes a rather long time, maybe a second or two each and every time. However, once its running the compilation itself is very, very quick. So, with the help of ReSharper (CTRL + R + O is your friend), I merged all the code into a single project file. The results are impressive - almost 2 orders of magnitude improvement!

10 builds in 16 secs @ 1.6 secs per build

Ok, so now we are starting to see where the bottlenecks lie in our build process. Next, I’ll be looking at the impact of various drives (HDD, SSD and RAMDisks) on compilation times.

So, my current role has us dealing with a build, deployment and integration test cycle of about 2 hours. Given the 50-odd developers spread across 6-7 teams, this is not a particularly nice situation. Apart from it being a PITA, a feedback loop of this length results in plenty of negative consequences, a selection of which include:

• feature branching (esp to mitigate cross team dependencies during a release phases)
• developer multi-tasking (“I’ll just start this other thing while I’m waiting for a build… oh wait now I’m doing 3 things at once”)
• difficult tracking build breakages (30 min builds mean commits bunch up - which one broke the build?)
• large batching of code changes (“if it takes 2 hours to get feedback, I’m going to save integrating my work until its finished”)
• forgetfulness (“I can’t remember what changes I made 2 hours ago that just broke the build”)

We get into a situation where we are no longer “continually integrating” our work, but moving back to the dark old days of hourly or nightly builds (or not at all!). Instead of speeding up, the teams are slowing down, releases become less frequent and quality inevitably falls.

So, considering the amount of lost development resource this was costing the company (2 hours times N developers = a lot of money and more importantly time), I was tasked with improving the situation in any way that we could. Some outcomes were possible, some not, some resulted in great improvements, others only minor gains. But it did mean that I did possibly more research into the MsBuild system, csc.exe, .NET assembly structures and Visual Studio project files than is probably healthy, in an effort to squeeze every last second of time out of the process.

I’ll present my findings, and a bunch of interesting articles I (re)discovered along the way in the next few weeks. In the meantime, and to help set the scene somewhat, enjoy this little gem from the man himself - Patrick Smacchia (creator of NDepend)

http://www.simple-talk.com/dotnet/.net-framework/partitioning-your-code-base-through-.net-assemblies-and-visual-studio-projects/

I’ve had to setup my dev environment on a few different machines recently. Quite a simple, but repetitive exercise, so I decided to see if I could make it completely portable. The TL;DR of it is that you can’t completely - but you can to a certain extent.

Here’s what I’ve managed to do so far: 

a) File sync

I’m using SpiderOak currently. DropBox only lets you backup a single folder, and Ubuntu One’s conflict resolution leaves a lot to be desired on Windows (at the moment) - appending .u1.conflict to filenames all over the place. The SpiderOak client is not perfect either - but it seems to be a workable solution so far.

b) Dotfiles

I’ve added some of my dotfiles to their own directory (.bashrc, .bash_history, .ssh/, .gitconfig etc), and got them syncing to the “cloud”. You can then use mklink to make a Windows version of a symlink, thus:

mklink /H ..\.bashrc dotfiles\.bashrc

c) Portable apps

Most of the tools I use have a portable version (obviously with the exception of Visual Studio & Resharper), so get these sync’d up as well

- Git (make sure you set GITDIR in your environment)

- Console2 (delete the config in your %APPDATA%)

- Sublime (this is a great little text editor I’ve just discovered, so long Notepad++!)

- WinSplit Revolution

- Unlocker

d) The extras

Next thing I did was to create batch files to set your symlinks and path. The symlinks work a treat, but the ENV/PATH stuff is a bit of a pain (more on this below).

As a nice to have, there are some registry files to add stuff to your context menu as thus:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\*\shell\Open with Sublime\command]
@="C:\\Users\\matt.jackson\\dev\\Sublime Text 2 Build 2139\\sublime_text.exe %1"

d) What I haven’t done yet

Git Extensions. There is a portable version I believe, but the Explorer menu is the main reason I use it. The GitExtensions project sets these via a MSI, which is less than ideal - I’m open to suggestions.

p4merge. Probably the best thing to come out of Perforce, and its free! Doesn’t appear to be portable though. 

Resharper license/settings. I know settings are stored in %AppData%, so this is on my TODO list. License information is probably in the registry but to be confirmed. Work in progress.

Autorun scripts on logon. Maybe add something to my .bashrc or similar that ensures everything is set correctly. Need to think on this more.

Add directories to your user PATH permanently. I could have sworn there was a way do this via SET but I still can’t find it. For now, its a case of copy/pasting into a dialog box.

I’d love to hear about people’s experiences with this. How far have you got and have you solved anything on the TODO list above?

Sonar is a shiny dashboard for code quality that provides an integration point for various quality frameworks (FxCop, StyleCop, NCover, Gallio, Gendarme etc). Unfortunately (like most awesome tools), its native to the JVM but there are actively maintained plugins for .NET.

So here’s a quick rundown for how I got it running on my local machine:

1. Install Sonar - http://docs.codehaus.org/display/SONAR/Install+Sonar
   1. Install requirements
      1. Make sure java.exe is in your search PATH (C:\Program Files (x86)\Java\jre6\bin)
      2. Tomcat - run as exe or install as service
      3. DB not required for demo but would for proper installation (MSSQL, MySql, Postgres etc)
   2. Unzip (c:\sonar)
   3. Run via c:\sonar-2.11\bin\windows-x86-32\StartSonar.bat
   4. Wait a few minutes for the JVM to spin up and create a local DB (/data)
   5. You can watch this via /logs/sonar.log
2. Install Sonar-runner (so we can run it without Maven) - http://docs.codehaus.org/display/SONAR/Analyse+with+a+simple+Java+Runner
   1. Set your JAVA_HOME (C:\Program Files (x86)\Java\jre6)
   2. (Optional) Copy the jdbc settings from sonar.properties to sonar-runner.properties

      1. sonar.jdbc.url: jdbc:derby://localhost:1527/sonar;create=true
         sonar.jdbc.driverClassName: org.apache.derby.jdbc.ClientDriver

   3. Add C:\sonar-runner-1.1\bin to your PATH
   4. To test: sonar-runner -h
3. Follow the C# plugin walkthrough here: http://docs.codehaus.org/display/SONAR/C-Sharp+Plugins+Ecosystem
   1. Download http://docs.codehaus.org/download/attachments/201228384/CSharpPluginsEcosystem-1.1.zip
   2. Extract to C:\sonar-2.11\extensions\plugins
   3. (Remove gallio & fxcop jars unless you’ve installed their binaries separately)
   4. Restart Sonar
4. Create a sonar config in your .sln directory http://docs.codehaus.org/display/SONAR/2.+Configure#2.Configure-CreateaSonarfileforyoursolution
   1. Create sonar-project.properties & paste skeleton config
   2. cd to your .sln directory
   3. Run sonar-runner from cmd
5. Bask in the shiny graphs

Some interesting things I’m still to do…

• Add Gallio with (PartCover|NCover) for unit test coverage
• Figure how to integrate it into the CI process
• “Stop the line” on certain metrics?

I’ve had to solve some interesting problems over the last few weeks, and although it doesn’t really surprise me, git has come to the rescue in every single case.  

Merging two git repositories

Say you were working on a spike of some independent code in its’ own repo, and you’ve decided you want to merge it into your main repository. Easy:

http://nuclearsquid.com/writings/subtree-merging-and-you/

Triggering TeamCity builds directly from github

GitHub is awesome. You should host your code there. But triggering your local build without polling them every 30 seconds? No problem:

http://www.jaxzin.com/2011/02/teamcity-build-triggering-by-github.html

Splitting a git repository (including history)

Finally, imagine a folder that you’d love to get rid of out of your repository. Hypothetically, let’s say a folder full of documentation, specifications, Word files, Visio files and the like. Hundreds of megs and you don’t really need that shit in a codebase do you? Well, let’s split it out:

http://stackoverflow.com/questions/359424/detach-subdirectory-into-separate-git-repository

// copy your repo to a new directory
git clone --no-hardlinks ./original_repo ./new_repo
// now filter out the folder(s) you want to keep
cd new_repo/
git status
git filter-branch --subdirectory-filter ./the_folder_you_want_to_split/ HEAD
git reset --hard
rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --aggressive --prune=now
// and head back to the original repo and delete that folder
cd ../original_repo/
git filter-branch --index-filter "git rm -r -f --cached --ignore-unmatch the_folder_you_want_to_split" --prune-empty HEAD
git reset --hard
git gc --aggressive --prune=now

And one for the road, here’s a cool little site that outlines what goes on under git’s covers:

http://think-like-a-git.net/

I did have a few free ebooks on the topic, I’ll see if I can dig them up - they are great reads.

Some little gems from my .gitconfig…

p4merge with git

[diff]
	tool = p4merge
	guitool = p4merge
[difftool "p4merge"]
	path = C:/Program Files/Perforce/p4merge.exe
	cmd = \"C:/Program Files/Perforce/p4merge.exe\" \"$LOCAL\" \"$REMOTE\"
[merge]
	summary = true
	tool = p4merge
[mergetool "p4merge"]
	path = C:/Program Files/Perforce/p4merge.exe
	keepBackup = false
	cmd = \"c:/Program Files/Perforce/p4merge.exe\" \"$BASE\" \"$LOCAL\" \"$REMOTE\" \"$MERGED\"

pull with automatic rebase

I have just discovered the beauty of pull with rebase (single, easy to read history on master! no more merges!). However, its a bit of a pain to continually do this:

git pull --rebase origin master

So to switch on rebase by default…

git config --global branch.autosetuprebase always

Beware! This only applies to new repo’s, not existing ones. To update existing repo’s with this setting try this:

git config branch.branch-name.rebase true

eg.

git config branch.master.rebase true

(reference: git with automatic rebase)

ssh-agent on windows

Ok, this one is just stolen straight from the github site here, but it still bears mentioning anyway. Use ssh-agent to hold onto your keys instead of stupid pageant. Add this to your .bashrc and have your keys loaded the first time you launch git bash after a boot!

SSH_ENV="$HOME/.ssh/environment"

# start the ssh-agent
function start_agent {
    echo "Initializing new SSH agent..."
    # spawn ssh-agent
    ssh-agent | sed 's/^echo/#echo/' > "$SSH_ENV"
    echo succeeded
    chmod 600 "$SSH_ENV"
    . "$SSH_ENV" > /dev/null
    ssh-add
}

# test for identities
function test_identities {
    # test whether standard identities have been added to the agent already
    ssh-add -l | grep "The agent has no identities" > /dev/null
    if [ $? -eq 0 ]; then
        ssh-add
        # $SSH_AUTH_SOCK broken so we start a new proper agent
        if [ $? -eq 2 ];then
            start_agent
        fi
    fi
}

# check for running ssh-agent with proper $SSH_AGENT_PID
if [ -n "$SSH_AGENT_PID" ]; then
    ps -ef | grep "$SSH_AGENT_PID" | grep ssh-agent > /dev/null
    if [ $? -eq 0 ]; then
	test_identities
    fi
# if $SSH_AGENT_PID is not properly set, we might be able to load one from
# $SSH_ENV
else
    if [ -f "$SSH_ENV" ]; then
	. "$SSH_ENV" > /dev/null
    fi
    ps -ef | grep "$SSH_AGENT_PID" | grep -v grep | grep ssh-agent > /dev/null
    if [ $? -eq 0 ]; then
        test_identities
    else
        start_agent
    fi
fi