Building a Development Environment Part 3: Managing the buildprocess Nant and Visual Studio

This is part 3 in my series on managing the build process.

What have we got till now? Well, we’ve setup a source code repository in Part 1 and setup CruiseControl.NET to monitor it in Part 2. But we wouldn’t need all this if we didn’t want to develop some software and thus have some source code we want to maintain and build.

That is what we will do in this part of the series: Create a simple library written in C# and build it automaticaly.

But first:

Disclaimer

This post is by no means “The Way” of building a development environment. In fact, you will notice that at times I deviate from some of the practices advocated by the referred articles at the end of my articles. It’s our interpretation of the process and how we implement it. If you have any remarks or suggestions for improvement, please post a comment.

Tooling used

Appart from the tooling of the previous articles, we also use

  • Visual Studio.NET 2005: This is run on the developer machines and is used to, well, write software.

Setup the developer machine

Remember how in the first post of this series we’ve setup some folders to mimic multiple computers? Well, now we will create another folder to mimic the developer workstation.

Organising our experimentation environment

In our simulation subfolder we create subfolders “wks1” and “wks2”. We will not be using the second one for now as we will first setup an environment for one developer.

So our simulation subfolder now looks like this:

simulation2.gif

The folders starting with wks represent developer WorKStations.

Organising the developer workstation

On the developer workstation we have a central folder where we do all our development. This folder is typically called “work” and further on we will refer to it as “the workstation’s working folder”.
Inside this working folder we have two subfolders:

  • Devenv: this folder is the same for all developer workstations and contains our buildscripts, third party applications used for developing software, etc…
  • Project3: This is a sample projectfolder and has the name of the project. It contains the sourcecode for that project. So it will have the three main folders that we use in our development environment: core, unittest and product. These are obtained by performing a check-out of the Project3 codeline in our subversion repository.

Create a module

Get our project from the Source Control Server

Before we create a module, we want to check out the codeline of the project we are working on. To do this, we first create a folder with the name of the project we are going to work on in our workstation’s working folder. Because we will work on a project called Project3, we will call our folder “Project3”. Next, we check out our Project3 codeline from our subversion repository. We simply right click on the folder in windows explorer and select the TortoiseSvn command “SVN Checkout…” We get the following dialog box:
checkoutproject.gif
In the “URL of repository” edit box you select the Project3 code line in the repository and in the “Checkout directory” edit box you select the freshly created Project3 subfolder in your workstation’s working folder. Click OK and watch TortoiseSVN get the code from your repository.

At the end we get the following folder structure, on our harddrive:
workingfolder.gif

Creating our library

As we’ve said in Part 1 of this series, we organize our code in modules. These modules map to Visual Studio project files (file extension csproj for C# projects). A product is build by grouping modules together. We name these groups “Components”. These components map to Visual Studio solution files (file extension sln). A product can of course contain several components, so our Product subfolder can contain several solution files. This can for example be the case of a product exists of several executables.

To create a product, we create a subfolder with the name of our product in the project’s Product subfolder. Then, to build a component, we create a solution in the product folder. Make sure you override the default folder of Visual Studio, because this creates another subfolder which we do not use.

Next we create a library in this solution. Again, Visual Studio creates a subfolder with the name of our library. So the complete structure of our folders is like this:

workingfolderwithmodulev4.gif

Build it!

Now, hit the compile button. This should not give any problems as we just have an empty class created by the wizard.
Ok, so we have some code that we can compile. It is created in a module which is in a component which is available for use in our products. But of course, we want to build products and not components or modules.

So, we could keep a file with the components used by our product, and manually build every component. This would of course be errorprone and result in components not being build. And mind you, allthough our current build is only compiling the sources, a real build, as you will see throughout the rest of this series, is more then just compilling the sources. It is also about executing unittests, static code analysis, generating code metrics, generating documentation, deploying binaries, releasing and tagging in the source code repository, etc…

Thus, what we really want is an automated repeatable build proces. Enter Nant…

How do we build manually?

If we would build manualy, here is how we would proceed:

  1. Is this a main or a project build? Select the correct subfolder
  2. What is the product? Select the correct product folder
  3. What components is the product made of? For each component…
    1. What modules is the component made of? For each module
      1. Build the module

So, our automated process must perform the same steps.

Build the module (of the current component, of the current project)

To automate the building of a single module we use the commandline parameter options of Visual Studio.

What Are You Doing ???!!!

I know, this is NOT DONE. Some CI purists will shoot me for this, but we are doing it anyway, and here is why:
We actually started implementing all this with Visual Studio 2003. So there was no MSBuild which is why everything uses Nant (appart of the fact that I wanted to use open source tools as much as possible). We also do not only use the .Net Framework, but also have a lot of C++ code? This last code can not be compiled by Nant.
So we where left with two options:

  1. Compile the .Net code with the <solution> task of Nant and the C++ code with the commandline version of Visual Studio.
  2. Unify our way of compiling and use the commandline version of Visual Studio to build all our projects.

The second option seemed the most logical to me.

To build a project inside a solution we use the following commandline:

[VisualStudioInstllationFolder]/devenv.exe 
   [PathToSolutionFile] 
   /Build [SolutionConfiguration] 
   /Project [PathToProjectFile] 
   /projectconfig [ProjectConfiguration] 
   OUT/ [PathToLogfile]

In our context it can be that we do not want to build every project in a solution, but only some particular projects.

So we need a way to specify:

  1. The Visual Studio installation folder
  2. The solution file: the solution filename is always the name of the solution itself, with extension .sln
  3. Where to look for the solution: the solution file is always in a subfolder Product/[ProductName] of the projectfolder (see the above folderstructure). This projectfolder is either:
    1. A subfolder of the workstation’s working folder with the name “Main” if we are working in the main codeline
    2. A subfolder of the workstation’s working folder with the name of the project if we are working in a project codeline
  4. The projects in the solution we want to build
  5. The project files in the solution and where they reside
  6. A filename for the logfile and where to put it

To do all this we first make a Nant file with global properties describing the various folders in our buildenvironment.
On my computer this becomes:

<?xml version=1.0?>

<project name=Basic settings>

  <description>Contains the basic settings for

    an installation of the PVS build environment.

    </description>

 

  <property name=dir.main

            value=${environment::get-variable(‘BUILDENV_WKS_WORKFOLDER’)} />

  <property name=dir.buildenv

            value=${dir.main}Devenv\ />

  <property name=subdir.core

            value=Core\ />

  <property name=subdir.product

            value=Product\ />

  <property name=subdir.log

            value=buildenv.log\ />

 

  <!– Folder where your Visual Studio.NET

        console executable resides –>

  <property name=dir.msvsnet

            value=D:\Program Files\Microsoft Visual Studio 8\Common7\IDE\ />

</project>

This is saved in a file buildenv.build which we save in subfolder “Devenv” of our workstation’s working folder. (This must be the same folder as pointed to by the dir.buildenv property in the above buildfile).

As you can see from the above, we use an environment variable to save the workstations working folder. To do this create an environment variable with the name BUILDENV_WKS_WORKFOLDER and set it to the value of your workstations workingfolder. In our case, we have:

envvariable.gif

To build a given list of projects in a solution, we have two global tasks:

  1. A task which uses a comma seperated list of projects in our solution which have to be build and for each projects in this list, calls a task which builds the project
  2. A task which builds a project using Visual Studio

For this we make a Nant buildfile called defaulttasks.build which also gets stored in the folder pointed to by the dir.buildenv property and has following content:

<?xml version=1.0?>

<project name=module tasks basedir=.>

  <description>Nant buildfile for modules.

    </description>

  <target name=productbuild

          description=build the core project>

    <foreach item=String

            in=${product.components}

            delim=,

            property=component.name>

      <echo message=building component ‘${component.name} ${product.configuration}’ />

      <property name=component.buildfile

                value=${project.basefolder}${subdir.product}${product.name}\${component.name}.build />

      <nant buildfile=${component.buildfile}

            target=buildcomponent />

    </foreach>

  </target>

  <target name=componentbuild

          description=build the core project>

    <foreach item=String

            in=${component.modules}

            delim=,

            property=component.module.name>

      <echo message=building module ‘${component.module.name} ${component.configuration}’ – logfile:${project.basefolder}${subdir.log}${component.name}\${component.module.name}.${component.configuration}.log />

      <property name=devenv.commandline

                value=“${component.dir}\${component.name}.sln” /build ${component.configuration}  /project “${project.basefolder}${subdir.core}${component.module.name}\${component.module.name}.csproj” /projectconfig ${component.configuration}  /OUT “${project.basefolder}${subdir.log}${component.name}\${component.module.name}.${component.configuration}.log” />

      <echo message=${devenv.commandline} />

      <exec program=devenv.exe

          basedir=${dir.msvsnet}

          commandline=${devenv.commandline} />

    </foreach>

  </target>

</project>

To build our product we have a buildfile with the name of our product which basically iterates through a list of components which have to be build and calls the buildfile for each component. This file is placed in the products folder. In our case it is called “SomeProduct.build”
This is what it looks like:

<?xml version=1.0?>

<project name=SomeProduct basedir=. default=executeall>

  <description>Buildfile for SomeProduct.

    </description>

  <include

    buildfile=..\..\..\Devenv\buildenv.build />

  <!– Edit following properties for your project –>

  <property name=product.name

            value=SomeProduct />

  <property name=product.configuration

            value=debug />

  <property name=product.components

            value=Component1 />

  <property name=project.basefolder

            value=${path::get-full-path(‘./../../’)} />

  <!– End editable area –>

 

  <target name=buildproduct>

      <nant

      buildfile=${dir.buildenv}defaulttasks.build

      target=productbuild />

  </target>

</project>

And each component has a buildfile which iterates through all the modules which have to be builb. The component buildfile has the same name as the component and is saved in the components folder. In our case it is called “Component1.build”.
This is what it looks like:

<?xml version=1.0?>

<project name=Component1 basedir=.>

  <description>Buildfile for Component1.

    </description>

  <include

    buildfile=..\..\..\Devenv\buildenv.build />

  <!– Edit following properties for your project –>

  <property name=component.name

            value=Component1 />

  <property name=component.thisconfiguration

            value=debug />

  <property name=component.modules

            value=ClassLibrary1 />

  <!– End editable area –>

  <property name=component.configuration

            value=${product.configuration}

            if=${property::exists(‘product.configuration’)} />

  <property name=component.configuration

            value=${component.thisconfiguration}

            unless=${property::exists(‘product.configuration’)} />

  <property name=component.dir

            value=${path::get-full-path(‘.’)} />

 

  <target name=buildcomponent>

      <nant buildfile=${dir.buildenv}defaulttasks.build

        target=componentbuild />

  </target>

</project>

So, this is the structure of our Nant buildfiles:

nantstructurev2.gif

  1. The SomeProduct.build file includes the Buildenv.build file with the global folders of our build environment
  2. The SomeProduct.build file calls the “productbuild” task in the defaulttasks.build file, giving it the list of components to build by a &li;property> called “product.components”
  3. The “productbuild” task in the defaulttasks.build file calls the “buildproject” task in the Component1.build file
  4. The “buildproject” task in the Component1.build file finally calls the “componentbuild” in the defaulttasks.build file giving it a list of modules to build by a <property> with the name “component.modules”. This “componentbuild” task finally calls Visual Studio for module (= project) in the component (= solution)

Next, to easily build our project, we create a simple batch file with the name “SomeProduct.bat” in our product folder. It has following content:

START /B ./../../../Devenv/Nant/bin/nant.exe -buildfile:SomeProduct.build buildproduct

Now, open a DOS window in the products folder and run SomeProduct.bat. You should see some Nant activity and finally the conclusion BUILD SUCCEEDED.

Setup CruiseControl.NET

Normally, the next step is to checkin our code in our repository, but now I will first show you how to setup our build server to automaticaly build our project when we checkin our changes in our repository.

In Part 2 we setup CruiseControl.NET simply to see if everything went well. We did not however setup a real project. That is what we are going to do now.

A folder structure for our projects

buildsrvrfolder.gif

In de workingfolder of our buildserver we make a subfolder with the name of our project, in our case “Project3”. In this folder we make a subfolder “Source” in which we will basically have the same structure as a developer workstation’s working folder. So make a Denenv folder in which to put the buildend.build and defaulttasks.build files and checkout the sourcecode of our project in a subfolder “Project3”, just like we did on the developer workstation.

WARNING: Edit the buildenv.build file on the CI server so that the property “dir.main” has a value “${environment::get-variable(‘BUILDENV_CI_WORKFOLDER’)}Project3\Source\”. Then, make an environment variable with the name “BUILDENV_CI_WORKFOLDER” and set it to the buildserver’s working folder, which in my case is “D:\Serge\Devenv\simulation\buildsrvr\”

We will also need a configuration file for CruiseControl.NET: the ccnet.config file. This will be saved in the buildservers working folder and looks like this:

<cruisecontrol>

  <project name=Project3>

    <workingDirectory>D:\Serge\Devenv\simulation\buildsrvr\Project3\WorkingDirectory</workingDirectory>

    <artifactDirectory>D:\Serge\Devenv\simulation\buildsrvr\Project3\Artifacts</artifactDirectory>

    <webURL>http://localhost/ccnet/</webURL>

    <triggers>

      <intervalTrigger seconds=60 />

    </triggers>

    <modificationDelaySeconds>10</modificationDelaySeconds>

    <sourcecontrol type=svn>

        <trunkUrl>svn://localhost/Projects/Project3</trunkUrl>

  <username>autobuild</username>

  <password>autobuildpwd</password>

        <autoGetSource>true</autoGetSource>

        <workingDirectory>D:\Serge\Devenv\simulation\buildsrvr\Project3\Source\Project3</workingDirectory>

    </sourcecontrol>

    <tasks>

      <nant>

        <executable>D:\Serge\devenv\devenv.software.external\nant-0.85-rc3-bin\nant-0.85-rc3\bin\nant.exe</executable>

        <baseDirectory>D:\Serge\Devenv\simulation\buildsrvr\Project3\Source\Project3\Product\SomeProduct</baseDirectory>

        <buildArgs></buildArgs>

        <buildFile>D:\Serge\Devenv\simulation\buildsrvr\Project3\Source\Project3\Product\SomeProduct\SomeProduct.build</buildFile>

        <buildTimeoutSeconds>300</buildTimeoutSeconds>

        <targetList>

          <target>buildproduct</target>

        </targetList>

      </nant>

    </tasks>

    <publishers>

      <xmllogger />

    </publishers>

  </project>

</cruisecontrol>

Allthough in this file it is specified that CruiseControl.NET automatically retrieves the changes from the sourcecontrol server, the first time you have to perform a check-out yourself. So, go to the %BUILDENV_CI_WORKFOLDER%Project3\Source\Project3 and do a check-out is you did on the developer workstation.

If everything was done right, you get the following folderstructure:

cistructure.gif

The Devenv folder contains the buildenv.build and adapted defaulttasks.build files.

Check-in our changes

Some preparation

Before we commit our changes to our repository, make sure you have two programs running:

  • The Subversion server: we need this one to be able to check in our sources
  • The CruiseControl.NET server: we need this one to monitor our checkins and perform an automatic build

Save it !

Now, to save our changes in our source code repository we do the following:

Right click the Project3 folder on our developer workstation and choose the TortoiseSVN > Add… submenu. This will present you with a dialog box with a summation of all the files in the Project3 folder and subfolders which have not been added to your repository yet.

addtorepository.gif

In our case these are all the folders and files just created and some supplementary which result from building our project. We do not want to add these last ones because we do not save build results in the repository, just our sources.

The files and folders we want to eclude are following:

  • Files with extension “suo”
  • Subfolders of our module folders with name “bin” and their content
  • Subfolders of our module folders with name “obj” and their content
  • Subfolders of our module folders with name “buildenv.log” and their content

Unselect these files and folders. Now click the OK button.

Excluding files from our repository
We have just unchecked the files and folders we do not want added to our repository manualy. But when we add new classes to our module, or modules to our component, each time we choose to add these files we will have to uncheck the type of files and folders from the list proposed by TortoiseSVN.

It would be more convenient if we could exclude them once and for all. Fortunatly, Subversion allows us to do this.

For a detailed discussion on how to ignore files and folders for inclusion in a Subversion repository, you can read my How To … article

For example, to ignore suo files in the product folder, you would perform following command in the parent folder of the component’s folder (this is typicaly the Core folder):

D:\Devenv\simulation\wks1\work\Project3\Product>svn propset svn:ignore *.suo SomeProduct

To ignore multiple types of files and folders, you must have a newline delimited list of values for the svn propset command. Because this can not be done with the commandline (well, I do not know how to do it anyway), we create a text file with on each line the signature of a file or folder to ignore.
Thus we have a textfile with following content:

obj

bin

buildenv.log

We save this file in the folder of the component in name it “ignore.txt”, and then issue the command in the component’s folder

D:\Devenv\simulation\wks1\work\Project3\Core>svn propset svn:ignore -F ignore.txt ClassLibrary1

I’ve choosen to set all the ignores using the local excludes. These get set on the repository itself and thus are also effective on other clients. With the global excludes you must adapt the config file on each client accessing the repository.

Also perform an add command for the Product folder. In this folder we include all the files.

Now, our local subversion client knows about these new files, but they have not been added to repository yet. For this you need to commit your changes by selecting the Commit menu in he TortoiseSVN submenu.

Automated Build

If everything went well, you should be able to see the buildresults when opening your webbrowser and go to the CruisControl.NET webapplication at the URL (at least on my computer) http://localhost/ccnet/

cibuild.gif

Conclusion

Well, we checked in some sourcecode and where able to automatically build it. Next we will extend the process with unit testing.

Resources

[1] Visual Studio 2005 Command Line Switches.

Updates

27 November 2006: original version
2 December 2006:

  • Provided some more detailed instructions on ceertain steps
  • Adapted Nant scripts to use environment::get-variable() for retrieval of the working folder
  • Changed location of log dump file when building a project with Visual Studio
  • Included description on how to exclude folders and files from addition to a Subversion source repository

10 januari 2007:

  • adapted to reflect the changes made to the Part 1 article.
  • removed part with detailed explanation about ignoring files in subversion and described it in a seperate post.
  • changed scripts so that the environment variables are no longer necessary
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s