Steve Eichert

What is the fastest way to dynamically instantiate an object?

Referenced Assemblies with CodeDOM

Last night I was working on a re-write of the base framework I've been using on a lot of my projects recently. Part of the re-write involved creating dynamic assemblies using CodeDOM. I made some good progress but ran into a problem when trying to reference assemblies. I used the CompilerParameters ReferencedAssemblies property to add each assembly but this didn't do the trick. When compiling the "code" I received a "Metadata file XXX could not be found." error. The dynamic assembly is created by the MyFramework assembly and is having problems finding the assembly of the MyProject assembly

• Customer object in MyProject calls a method on the object its inherited from (AbstractEntity) which is in MyFramework
• The AbstractEntity object in MyFramework should (if things weren't broken) create a dynamic assembly to perform some operations on the Customer object in MyProject when the method is called.
• When AbstractEntity tries to create the dynamic assembly it adds references to MyFramework and MyProject using CompilerParameters.ReferencedAssemblies.Add()
• A "Metadata file MyProject.dll could not be found" error is returned when compiling.

Any ideas?

RE: Fixed Bid Follies

Jonathan Goodyear just posted a very interesting article about stearing clear of fixed bid projects.  I've been thinking about this a good bit recently due to some of the outcomes of some fixed bid projects I've worked on in the past couple of months.  It seems that "we" (as service providers) too often force ourselves into fixed bid projects rather then trying to educate the client on how an hourly contract may be more appealing for both parties.  

In a post last month I talked about Agile development and how it can be applied to fixed bid projects.  I've come to the conclusion that it is possible to use Agile methodologies on fixed bid projects but doing so requires a lot more un-agile practices to be used in order for it to work.  I still do almost all of my side projects on fixed bid contacts but they almost all fall into the under 200 hour limit Jonathan spoke about in his article.  I'm slowly coming to the realization that I need to start changing my ways in order to provide myself and my clients with the greatest chance of success.  It isn't going to be easy but in the long run I'm sure I'll be more successfull because of it.

Is Storing images in SQL Server a good or bad idea?

Update: Check out the final results !

I just started on a little side project that is going to allow an administrator to upload documents through an administration tool.  On the public side of the site users will login to a members only area and be able to browse the documents.  I've decided to see how storing files in SQL Server will work.  I figure this is a good project to try it out on since its a pretty small app with a low user base.  I like the idea of not having to worry about configuration on the server so we'll see how it goes.  It looks like I'll have to alter some of my framework code to work with the byte[] data type. 

Anyone feel the urge to share any experiences with storing files in SQL Server?

What would my ideal entity object framework look like?

Let me start with a definition of what I'm calling an "Entity Object Framework".  In the world of object oriented developement we often create a set of objects to represent the "business entities" that make up our application.  These business entities have properties and methods that define who they are and what they do.  In almost every application these business entities need to be persisted to some sort of data store.  An "entity object framework" provides the plumbing for dealing with entities, as well as their persistence.  Rather then worrying about how to persist the business entities the developer should only worry about ensuring all the properties, methods, and object relationships are defined properly on the object.  The developer shouldn't have to write redundant code for persisting each type of business entity, it should all just work through the magic of the entity object framework.

What I'm calling an Entity object Framework may sounds very much like an Object Relational mapper.  In fact perhaps some day one of the OR Mappers (EntityBroker, DeKlarit, etc) will provide me with just what I'm looking for. 

In my ideal world I want to deal with objects, not DataSets, and not Xml.  Now that is not to say that the EOF shouldn't have support for DataSets or Xml, its just that in my world I want to choose to use them when and where they make sense for me, and the application I'm working on.  I must state that I haven't really given a pure Xml or pure DataSet (typed) based architecture much of a chance so perhaps that should be something I put on my long list of todo's.

When I'm working with my entity objects I want to be able to use a .Save(), .Delete(), .GetAll(), .GetByID() and etc methods directly from my objects.  I don't want to have to go through a data manager class, I don't want to go through a broker, I just want the objects (actually the framework) to handle these operations for me.  In my ideal world I want the object to not only handle it for me but to handle it for me without forcing me to write code.  I should be able to provide some meta data through some custom attributes that will allow my objects to figure out how to save themselves, figure out how to load themselves, and figure out how to give me all objects where a certain property equals a value I'm interested in.  I haven't totally figured out all the details of how a framework would handle all of this but I've figured out enough of it to know it can be done. 

Not only should my entity objects be able to figure out how to save, delete, load, and find themselves from my data store but they should also be able to be configured to use whatever data store I choose.  Whether it be in Sql Server, Oracle, Access, Xml, or Binary it should just work!  Additionally it should work as well as if we had custom written each implemention ourselves.  It should provides us with the same performance, features, and extensibility and should do so without us having to write any of the code.  Now let me quickly point out that I'm not suggesting that a framework would or could do everything for everyone without anyone ever having to write a peice of code.  My ideal framework would allow for new data store support to be added by creating a class that is smart enough to dynamically create an assembly for handling the persistence of our objects to the data store.  The dynamic assembly would implement an interface or inherit from an abstract class so that we could easily plug in new dynamic assembly creators to the framework.  This would allow developers to use whatever data store they like to store their entities.  And would make changing what data store an application uses as simple as switching a configuration setting.

Ok, so this is of course the ideal world.  There are many things that I'm not considering and many things that may not be totally realistic.  I do believe much of what I'm looking for in my EOF is very doable and very possible.  I don't however believe that a framework can be created to do everything I would ever want.  There is just too many unrealistic things I could think up that would make the framework go boom.  Oh well, thats why this is in my ideal world....

Dynamically generating assemblies

This is an except from some notes I put together while working on peices of the entity object framework I developed a couple months back...

During the initial design of the Entity Object Framework a focus was placed on creating a set of base classes to handle the core CRUD (Create, Read, Update, Delete) operations for persistent objects. 

The Reflective Approach

The initial implementation in the EOF used a combination of custom attributes, and reflection to persist objects.  By looping over all the properties of an object and checking for a SqlParameter attribute the framework could successfully create a SqlDb object with all the parameters necessary for saving the object to the data store. (See ReflectiveSave.cs)

While this method of automatically executing the proper SQL required for the save of the object performed the desired actions it didn' t seem like an optimal solution (although not bad for starters). 

After further analysis it became clear that the overhead associated with looping though all the properties of an object each time a save was performed was not going to cut it.  A more streamlined approach that only involved looping over the properties on an initial save would undoubtedly result in better performance.  The only question was how best to implement this approach& an xml file for lookups? ...some other caching mechanism?  & or better yet how about a dynamic assembly.

The Dynamic Approach

The .NET Framework provides two main options when it comes to creating dynamic assemblies.  The first option is to use the System.CodeDom classes to programmatically create code constructs, save the results to disk, invoke a compiler, and then dynamically load the assembly into memory.  The second more direct approach is to use the System.Reflection.Emit classes to create a dynamic assembly using Intermediate Language.

CodeDom -

The System.CodeDom Namespace provides several classes for creating C# or VB.NET source code.  Using theses classes a developer has the ability to dynamically create a source file.  Once the source file is generated one can invoke the necessary compiler to create an assembly from the source file(s).  After generating an assembly from the dynamically created source files the assembly can be loaded using Assembly.Load.  The advantages to this method are:

• The generated source file can be viewed
• Debugging is easier
• The method of generating the source file is more straight forward then other methods

The disadvantages of this method are:

• Dynamically generated source files may crowd up the directory
• The performance isn't as good as creating the assembly in memory using IL (based on research no actual test performed)
• Update: These are both invalid points, not sure where they came from :-)

Reflection.Emit -

The compilation of C#, VB.NET, Jscript.NET, Cobol.NET or any other language in the .NET framework results in the creation of Intermediate Language (IL).  When the application is executed the Just-in-Time (JIT) compiler converts the IL into machine code.  By dynamically creating an assembly in IL using the System.Reflection.Emit class we can create an in memory assembly for our intended need.  The advantages to this method are:

• The generated assembly can be optimized using specific IL instructions
• The performance of the dynamic assembly is VERY GOOD

The disadvantages to this method are:

• Debugging the dynamic assembly is very difficult
• There is limited documentation regarding the creation of dynamic assemblies using Reflection.Emit

To to create the dynamic assemblies for the EOF I chose to use the Reflection.Emit method.  The process of creating an assembly using this method required me to get familiar with IL, as well as the process for creating methods in IL.

Creating the IL

The C#, VB.NET, Cobol.NET, and all other compilers that support the .NET Framework generate IL when compiling a set of code.  To get an idea of what IL looks like, start up ILDASM by opening a Visual Studio .NET Command Prompt (Start -> All Programs -> Microsoft Visual Studio .NET -> Visual Studio .NET Tools -> Visual Studio .NET Command Prompt) and type ILDASM.  The ILDASM tool is an IL Disassembler for the .NET framework which is installed as part of the .NET SDK.  To view the IL for any .NET component on your system go to File -> Open and select a .NET assembly.

To get familiar with IL I wrote a simple assembly in C# that performed a set of operations.  The operations were modeled on the method design I had in my mind for the dynamic assembly.  Looking at the resultant IL allowed me to get a better understanding of the ways in which IL Op Codes where used, as well as how I would need to chain them together to get the desired result.

After getting fairly familiar with the IL that would be necessary for my objects I began the construction of my DynamicAssemblyManager class.  The DynamicAssemblyManager class needed to have the ability to create a dynamic assembly for saving objects, as well as have the ability to store the cached assemblies in an easy lookup.  To handle the creation of dynamic assemblies a CreateAssembly method was created.  The CreateAssembly method accepted an object and returned a dynamically generated assembly that could save any object of the type passed in.  A private hash table was added to the DynamicAssemblyManager class to allow a quick and easy lookup of already generated assemblies.

The final piece necessary to complete the implementation of the dynamic assembly persistence was in the entity object class.  A private DynamicAssemblySave method was added to the class which handles the process of checking the DynamicAssemblyManager class for the existence of a dynamic assembly, and if it doesn't exist, creating it.

See CreateAssembly.cs

See DynamicAssemblySave.cs

The Results

After all the coding was done it was finally time to test the different methods for persisting the object to the database.  In total we ended up with three types of methods for saving the properties of an object to the database.

Method #1:

Use reflection to loop through all the properties of an object and dynamically add the necessary parameters and parameter values to the SqlDb object to be saved. (See ReflectiveSave.cs)

Method #2:

Dynamically generate an assembly to handle the persistence of the object to the database. (See CreateAssembly.cs and DynamicAssemblySave.cs)

Method #3:

Write a custom method to save the object . (See CustomSave.cs)

To test the performance of each of the methods specified above a NUnit test was setup.  The test class looped 10,000/100,000/1,000,000 times and called each of the 3 save methods.  In order to test only the performance of the object operations and not the actual execution of the SQL call against the database the objects used a stub SqlDb object.

The results of the performance tests are detailed below:

First set of 10,000 repetitions

• Save w/ Reflection: 2094 ms
• Dynamic Assembly: 296 ms
• Custom Save: 266 ms

Second Set of 10,000 repetitions

• Save w/ Reflection: 1953 ms
• Dynamic Assembly: 281 ms
• Custom Save: 266 ms

First set of 100,000 repetitions

• Save w/ Reflection: 19,016 ms
• Dynamic Assembly: 2,625 ms
• Custom Save: 2,438 ms

 Second Set of 100,000 repetitions

• Save w/ Reflection: 18,672
• Dynamic Assembly: 2,641
• Custom Save: 2,390

First set of 1,000,000 repetitions

• Save w/ Reflection: 170,047 ms
• Dynamic Assembly: 21,688 ms
• Custom Save: 20,187 ms

Second Set of 1,000,000 repetitions

• Save w/ Reflection: BOOM!
• Dynamic Assembly: 21,687
• Custom Save: 20,657
   

As you can see from the performance tests above the dynamic assembly performed almost as well as the custom written save method.  The slight difference may be due to the generation of the assembly or due to casting involved within the dynamic assembly.  Some tweaks to the IL may help make this difference smaller or non existent.

References:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfSystemReflectionEmit.asp

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemreflectionemitopcodesclasstopic.asp

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfSystemCodeDom.asp

Quality Resources for Computer Programming

In Brad Abrams latest post he talks about Subclassing Usage Guidelines.  The post provides some nice insights into some of the things to consider when designing an object model.  In Brad's post he links to an article on the Liskov Substitution Principle.  The link brings us to WordenWare.  The site contains some very nice articles regarding OOD best practices.  The following articles were an excellent refresher on some of the OOD principles that we should try and keep in the forefront of our mind when designing object oriented systems.

• Design By Contact
• Liskov Substitution Principle
• Open-Closed Principle
• Dependency Inversion Principle

Agile projects continued...

Peter Provost had a couple of nice responses to some of the questions I asked in my two entries regarding using agile practices on fixed bid projects. 

In responding to my question about how to use agile practices on fixed bid projects Peter responds with:

Honestly, the answer is that you DON'T do fixed bid projects. At least not the way most people think of them. 

Peter goes on to explain that by addressing each feature individually, and having the client write Acceptance Tests it is possible to do a psuedo fixed bid project using agile methodologies:

If the customer is willing to provide good Acceptance Tests for all of the features selected in an iteration, then you can fix price each iteration before it starts. This is as close as you can get. You still get the agility of short controlled iterations and the customer gets to control their spending.

From the sounds of it Peter and I are on the same page regarding how best to work with a client that demands a fixed bid contract.  While all this sounds very good its important to realize that non-fixed bid projects don't come without their challenges. 

Is it possible to do fixed bid projects using agile methodologies?

Today Martin Fowler discusses using agile practices on a fixed bid project.  I've done a good bit of reading on this subject since my last post on the topic and I'm still not convinced it can work in most scenarios.  Using agile development on a fixed bid project requires that the client has a strong trust in you, and it also requires the client to understand the software development process. 

If the client doesn't trust you how will they ever agree to paying $x for an undefined deliverable?  Most clients want to know what they are getting for their money so saying "well we're not really sure what exactly your going to get" just won't fly.  If you have the trust of your client the next requirement is that they understand the software development process.  They need to understand that things will go wrong.  Problems will arise.  Adjustments will need to be made.  If they don't understand this they won't be very happy when features need to get cut to fit in with the fixed price as well as the deadline.

Now, lets assume for a second that you have the complete trust of your client, and they also understand the software development process.  If this is the case forget about doing a fixed bid project and educate the client on the benefits of an hourly contract .  Fixed bid project in general, and especially large fixed bid projects are extremely difficult to complete successfully.  Most go over budget and end with a bad feeling for the client as well as the service provider.

Testing: Knowing when to say when!

Over the past 6 months I've slowly incorporated test driven development (TDD) into each of the projects I've worked on.  In short, test driven development rocks!  TDD has improved the quality of my code as well as the quality of the deliverables I provide my clients.

The major benefits I've seen from using TDD are:

• The logic of the application is verified for accuracy.
• Errors are identified early in the process.
• Allows for regression testing when new features are added, or existing features are changed.
• During the early stages of development it helps verify the planned architecture.
• Ensures that objects are designed for their consumers.
• Helps to get the creative juices flowing during the early stages of development.
• Helps to identify areas of code that could benefit from refactoring.

While test driven development has many benefits it doesn't come free of charge.  Writing tests classes takes time.  Planning the strategy for testing the system takes time.  Each developer needs to take an oath to stick by the TDD methodology throughout the project, even if deadlines begin to loom.

In trying to stick to the TDD methodology I've seen developers get obsessed with making sure every method of every class has a corresponding test.  As developer's we need to know when to say when.  Testing every method of every class may sound good in theory but it can cause problems.

• It takes too long.
• It increases the amount of code to maintain.
• Makes the TDD process a burden for developers.

In order for TDD to work, it needs to stay a fun part of the development process.  If developers begin to view TDD as a burdensome task that some power hungry manager is forcing upon them it just won't work.  If a change to a 1 hour change in functionality causes the developer to spend another 4 hours updating tests it won't work.  Testing needs to be fun, it needs to get the creative juices flowing, it needs to be something developers want to do, not something they have to do!

Field length validator for textarea's

One of the bugs in the system I'm currently working on was due to a textarea allowing more text then it's associated database field allowed.  Since the MaxLength property of a multiline textbox doesn't actually do anything I decided to create a validation control so we could re-use the logic everywhere necessary.  The validation control is available here: FieldLengthValidator.

Most of the base functionality for the control is provided by the BaseValidator class.  The EvaluateIsValid method does the work of determining if the value for the ControlToValidate control is valid.  Finally a set of properties (MinLength, MaxLength) are used to set the length validation requirements.

I realized after writing the validator that I could have just used a RegularExpressionValidator (see below) to validate the length of the control but the control only took a minute to write and may be a little more straight-forward.

Example of using the FieldLengthValidator:

<CustControls:FieldLengthValidator MaxLength="255" runat="server" ErrorMessage="Please limit the length of the content entered to 255 characters." Text="*" ControlToValidate="Content"/>

Example of using RegularExpressionValidator:

<asp:RegularExpressionValidator ControlToValidate="Content" ValidationExpression=".{0,255}" Text="*" ErrorMessage="Please limit the length of the content entered to 255 characters." Runat="server"/>

Yukon & Whidbey

RE: Is Storing images in SQL Server a good or bad idea?

A couple of weeks ago I asked whether storing images in SQL Server was a good or bad idea.  I've just pushed the code live for the project I was working on and thought I would post about my experiences. 

The first step in the creation of the application was the design of the database.  The table to store the documents ended up much as you would expect.

CREATE TABLE [dbo].[Document] (
 [DocumentID] [int] IDENTITY (1, 1) NOT NULL ,
 [DocumentName] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
 [Description] [varchar] (1000) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
 [DocumentData] [image] NOT NULL ,
 [Filename] [varchar] (250) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
 [ContentType] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO

Next was the creation of a stored procedure for saving the documents to the database. 

CREATE PROCEDURE ap_SaveDocument
(
@DocumentID  int = null,
@DocumentName  varchar(50),
@Description  varchar(1000),
@DocumentData  image = null,
@Filename  varchar(250) = null,
@ContentType varchar(50) = null
)
AS

 SET NOCOUNT ON  
 if @DocumentID is null begin 
  
  insert into Document
  (
  DocumentName,
  Description,
  DocumentData,
  Filename,
  ContentType
  )
  values
  (
  @DocumentName,
  @Description,
  @DocumentData,
  @Filename,
  @ContentType
  )
  
  SET @DocumentID = SCOPE_IDENTITY()

 end
 else begin
 
  update  Document
  set  DocumentName = @DocumentName,
     Description = @Description
    where DocumentID = @DocumentID 
 
  -- update the document if provided
  if @DocumentData IS NOT NULL
   update Document SET
      DocumentData = @DocumentData,
      Filename = @Filename,
      ContentType = @ContentType
      WHERE DocumentID = @DocumentID
      
 end 
  
 RETURN @DocumentID

And finally the code to save the document.

ID = db.RunProc(SaveProc, sqlParams);
return ID > 0;
}

If your interested in seeing the entire Document class click here.  The final piece to the puzzle is the .aspx page for retrieving the document from the database and outputting it to the users browser.

Dealing with null database values

When working with data we often have to deal with the NULL condition within our code.  As an example lets look at the Products table in the Northwind database:

CREATE TABLE [dbo].[Products] (
 [ProductID] [int] IDENTITY (1, 1) NOT NULL ,
 [ProductName] [nvarchar] (40) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL ,
 [SupplierID] [int] NULL ,
 [CategoryID] [int] NULL ,
 [QuantityPerUnit] [nvarchar] (20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
 [UnitPrice] [money] NULL ,
 [UnitsInStock] [smallint] NULL ,
 [UnitsOnOrder] [smallint] NULL ,
 [ReorderLevel] [smallint] NULL ,
 [Discontinued] [bit] NOT NULL
) ON [PRIMARY]

Now lets assume we use the following code to retrieve all the products from the table:

// ensure a blank sa password for security :-)
SqlConnection conn = new SqlConnection("Server=(local);Initial Catalog=pubs;User ID=sa;Password=;");
conn.Open();
// setup command and retrieve a data reader
SqlCommand cmd = new SqlCommand("SELECT * FROM Products", conn);
SqlDataReader dr = cmd.ExecuteReader(CommandBehavior.CloseConnection);

The above code will give us a SqlDataReader object loaded up with all the product data from the Northwind database.  Looking again at the create statement for the Products table we can see that many of the columns are nullable.  I've seen a couple of different methods for dealing with the null condition, below is the most common method I've seen used:

// code to load a product collection with all the products in our data reader
ProductCollection products = new ProductCollection();
while(dr.Read())
{

Product product = new Product();
product.ProductID = Convert.ToInt32(dr["ProductID"]);
product.ProductName = dr["ProductName"].ToString();
product.SupplierID = dr.IsDbNull(dr.GetOrdinal("SupplierID")) ? -1 : dr.GetInt32(dr.GetOrdinal("SupplierID"));
product.CategoryID = dr.IsDbNull(dr.GetOrdinal("CategoryID")) ? -1 : dr.GetInt32(dr.GetOrdinal("CategoryID"));
// and so on for the remaining fields
products.Add(product);

}
dr.Close();

One of the members of our team developed a simple utility object for dealing with the null condition which removes some of the ugliness as well as duplication with the above method.  Rather then provide the entire method here I'll just provide the interface for the utility object as well as one example implementation of the RetrieveInt32 method.

/// <summary>
/// Retrieve a Int32 from a IDataReader
/// </summary>
/// <param name="dr">The IDataReader to read the string from</param>
/// <param name="columnName">The name of the column</param>
/// <param name="valueIfNull">The value to return if the column is null</param>
public static int RetrieveInt32(IDataReader dr,string columnName,int valueIfNull)
{

int retVal = valueIfNull;
try
{

retVal = dr.IsDBNull(dr.GetOrdinal(columnName)) ? valueIfNull:dr.GetInt32(dr.GetOrdinal(columnName));

}
catch (SqlException e)
{

throw new DataUtilityException(_dataUtilityExceptionUserMessage, "SqlException occured while retrieving the column:" + columnName, "RetrieveInt32", e);

}
catch (IndexOutOfRangeException e)
{

throw new DataUtilityException(_dataUtilityExceptionUserMessage, "The name specified is not a valid column name:" + columnName, "RetrieveInt32", e);

}

return retVal;

}

And finally a look back at the code to load the products using the the described utility class.

// code to load a product collection with all the products in our data reader
ProductCollection products = new ProductCollection();
while(dr.Read())
{

Product product = new Product();
product.ProductID = Convert.ToInt32(dr["ProductID"]);
product.ProductName = dr["ProductName"].ToString();
product.SupplierID = DataUtility.RetrieveInt32(dr, "SupplierID", -1);
product.CategoryID = DataUtility.RetrieveInt32(dr, "CategoryID", -1);
// and so on for the remaining fields
products.Add(product);

}
dr.Close();

As you can see the use of a utility class improves the readability of the code, reduces the amount of code required, and overall gives a much cleaner feeling.  Feel free to share your own solutions to dealing with the null condition as well as thoughts on the solution described in the comments for this entry!

Converting a "legacy" database to SQL 2K

In this evenings post I am going to overview the process I used on a recent project to convert a legacy database to SQL Server 2000.  The "legacy" database was an Access 95 database.  The database contained approximately 500 MB of data, 50+ tables, 200+ queries, and quite a mess of relationships.  The most interesting attribute of the database was that it was in French.  Considering I don't know a lick of French I'm sure you can imagine how much fun I had during the conversion.

The client was based in France and most (if not all) the users of the application were French speaking individuals so it wasn't that odd that the database was in french.  Fortunetely for me, we had an intern from France at the time who could help with the translations.  The first step in the conversion of the database fell in the lap of that lucky intern.  He went through each table and translated it to English.  After the tables were translated I wrote a simple script to generate insert statements to bring the data from the french database into the english database.  The script queried system tables in the database to construct the insert statements.  Although the generated scripts were not perfect they were certainly better then hand writing them myself.  Additionally some minor adjustments to the scripts allowed me to generate a set of documentation so when working with the client I could "translate" between the two database models.

After generating the insert scripts I created a DTS package to move the Access database to SQL Server.  Several of the tables contained invalid and/or incorrect data so I wrote a set of data cleansing scripts that cleaned out all the incorrect data.  Once the DTS package was successfully running we were able to transfer data from the Access database to the SQL Server database.  On top of having the data in SQL Server I was also finally able to look at the structure of the database and actually understand what data was being stored.

The 200+ queries wthin the access database was the final piece of the database that needed to be addressed.  Since the queries were primarily used by forms within the Access database they didn't provide a lot of value.  Rather then stressing over getting each of the queries converted I decided to write stored procedures on an as needed basis.  This provided me the ability to get familiar with the structure of the database and also ensure the queries were only returning the data necessary for the new app.  After evaluating what stored procedures the application required I again relied on the meta data available within the system tables to generate basic stored procedures for saving, deleting, and selecting data from the database.

Overall the conversion of the database was pretty straight forward.  The major take away items were:

• I don't know a lick of french
• Babelfish can be very useful
• Converting a database model to something you can understand is a good first step ;-)
• System tables provide all the necessary data required to generate a set of basic stored procedures for saving, deleting, and selecting data from a database.
• System tables can be helpful in auto-generating database documentation.
• Converting an Access database to SQL Server 2K is can be easy.

Project Overview

A provider of commercial real estate services wishes to convert a legacy application to the web.  The application allows real estate professionals to search a database containing 25+ years worth of data on various real estate properties. 

The primary objectives are to convert the legacy application's data store to SQL Server 2000, and provide a web interface for searching and reporting on properties within the database.  Once the application is converted the client will provide access to the application through a subscription service.

As stated, the datastore for the application will be converted to SQL Server 2000.  The application will provide all data access by way of stored procedures and will also make use of many of the XML features available within SQL 2k.

The middle teir components will be written using C# and will consists of data access components, proxy classes for web services, business objects, and utilitiy classes.

The front end will be built using ASP.NET, C#, Chart FX, and Active Reports .NET.  Additional components used in the development of the applicatoin include NUnit, NDoc, Nant, log4net and MapPoint.NET.

In the next couple of days/weeks I'll be discussing the following topics:

• The database conversion
• The architecture of the application
• Localization
• Creating reports using ChartFX and Active Reports .NET
• MapPoint .NET Integration
• Continuous Integration - Experiences with NUnit, Nant, and an overview of the build process

An introduction to a series of entries on something more interesting, well maybe

This evening I was sitting here catching up on some of the blogs I "subscribe" to and I started thinking about what makes for an interesting blog.  I haven't really come up with what it is yet, but when I began thinking about it I realized that mine probably isn't all that intersting.  I've talked about some of the entity object stuff I've worked on, put a couple tip like posts up, and rounded out the posting with random links and rants on miscellaneos stuff.  While I find the stuff I've done with the entity object framework interesting I realize its difficult to peak the interest of others without opening up the code for everyone to see. 

So, I began thinking about what I could write about that might be intersting to anyone who stumbles across one of my entries (hopefully more now that DNJ has that main feed going)...and then I thought some more since I didn't think of anything...and some more.....and then for some reason I gave up on thinking and opened up IE.  I then browsed to the homepage of a site I spent the better part of last year working on.  I logged in, started checking out how things were going, and thought...."Hey this was a pretty cool project to work on, with some really interesting twists and turns, and a pretty nice combination of components / technologies.". 

I can't talk about the project in detail but I can talk a lot about the trials and tribulations I went through during the project.  Will it be interesting?  Who knows.  At the very least it will give me a way to document what I did on the project, and hopefully learn something from putting my thoughts regarding the project in words.  Look for the project introduction shortly!

Higher Primates Can Program

First companies move jobs overseas and now this...   [Via Chris Sells]

A Googlewack-ish result?

I remember a while back I came across a site that lets you enter a Url to a rss feed and then see others who are linking to it.  In an attempt to find the site I went to google and entered "who is linking to my weblog". And the result you ask?  A googlewack-ish result.  Only one result (quotes disqualify it from being an actual googlewack, as well as more then two words) but none the less I didn't find anything to help. 

So, does anyone know what I'm looking for?  Say I want to see who is saying interesting things over at weblogs.asp.net, I should be able to enter the main rss feed as a starting address and see who is linking to individuals posts. 

UPDATE: http://www.technorati.com/ - thanks Darrell

NOTE: I did not initially actually read the criteria for what defines a googlewack so have modified this posting slightly.  That should teach me to assume I know what something is without doing a minimum amount of research!

Xml support for business objects

One of the advantages of using DataSets is their support for Xml.  By using the ReadXml() and WriteXml() methods on the DataSet developers can quickly retrieve an Xml representation of a DataSet, as well as re-hydrate a DataSet with data from an Xml file. 

By default our custom written business objects don't have support for such features.  The Xml Serialization features in the .NET Framework allow us to add Xml support to our objects with very minimal effort.  The code below shows an example implementation of a ToXml() method for returning an Xml representation of a business object, and a FromXml() method that returns an entity object re-hydrated with values from an Xml string. 

/// <summary>
/// Retrieve a Xml representation of the business object.
/// </summary>
/// <returns>String of xml representing the object.</returns>
public virtual string ToXml()
{

string xml;
System.IO.MemoryStream stream = new System.IO.MemoryStream();
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(this.GetType());
serializer.Serialize(stream, this);
 xml = System.Text.Encoding.UTF8.GetString(stream.ToArray());
stream.Close();
return xml;

}

/// <summary>
/// Re-hydrate an entity object from an Xml string.
/// </summary>
/// <param name="xml">The xml representation of the object</param>
/// <returns>The re-hydrated IEntityObject</returns>
public IEntityObject FromXml(string xml)
{

IEntityObject entity = null;
System.IO.StringReader reader = null;
try
{

System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(this.GetType());
reader = new System.IO.StringReader(xml);
entity = (IEntityObject)serializer.Deserialize(reader);

}
finally
{

if(reader!=null) reader.Close();

}

return entity;

}

What I'd like to be reading...

Unfortunetly I've been somewhat swamped recently with little projects that have prevented me from getting through some of the books I'd like to, anyway here's my list:

Currently Reading:

• Design Patterns
• How to Move Mount Fuji
• Software Project Survival Guide

Future Reading:

• Amazon Wish List (anyone want to help with prioritizing?)

Planned Re-Reads:

• Patterns of Enterprise Application Architecture

Recently Read:

• Agile Software Development, Principles, Patterns, and Practices
• Test Driven Development: By Example

C# Xml Commenting tip

Who is up for a "Don't ship shit manifesto"?

So far I think we have Roy, Lawrence, Dejan, Robert, any others? 

"I WILL NOT SHIP SHIT." 

- "I am a professional -- a craftsman!"
  -- "No matter what pressures are on me."
  -- "No matter how I've had to bend the rules."
  -- "No matter what shortcuts I've had to take."
  -- "No matter what the gods, or managers, have done or may do."

  --  --  "I WILL DO THE BEST WORK I CAN POSSIBLY DO."
  --  --  "Anything short of my best is shit."
  --  --  "I _ WILL _  NOT _ SHIP _ SHIT."

[Robert Martin]

Keep it simple!

I recently came across a post from Robert C. Martin titled Use Cases -- A minimalist's view.  Here is a quick excerpt from the post that hit home for me:

The real trick to use cases is to keep them simple. Don't worry about use case forms, just write them on blank paper, or on a blank page in a simple word processor, or on blank index cards. Don't worry about filling in all the details. Details aren't important until much later. Don't worry about capturing all the use cases; that's an impossible task.

What is a pattern?

Patterns gone awry

There has been a lot of hype in the .NET community recently regarding patterns.  Microsoft iteself has gone through a bit of a transformation from advocating a procedure based methodology towards a more OO, pattern oriented methodology.  This can be seen by the recent activity on MSDN, and more specifically in the Patterns & Practices development center.  Undoubtedly one of the side effects of all the hype will be developers attempting to use patterns without an understanding of their intended use, and without an understanding of how to implement them to solve their specific problems.

I recently was brought into a project to help resolve some issues as well as to implement some new functionality.  I quickly realized that my suspicions regarding "the hype" were indeed correct.  The developers working on the application had attempted to use patterns to solve some problems where the proper implementation of a pattern may have been a nice solution.  Unfortuntely the developers didn't really understand the patterns they were trying to implement.  This caused some instability in the application, and overall confusion. 

I consider myself to be far from an expert on patterns.  I've done a good bit of reading to get myself more familiar with patterns but still have a long way to go.  I think with a proper understanding of patterns developers can better attack many of the problems which we commonly face in building software.  I just hope we remember to be the wisest of men when going into battle.

Do you know your abstractions?

I've been monitoring Eric Sink's blog for a couple of weeks now but up until now I haven't really done any digging to see what content he put out before I came across his blog.  Today I came across The .NET Abstraction Pile and really enjoyed reading it.  He gives a real life example of how .NET kicked ass for his company.  His discussion of how many abstractions he dealt with during the development of Vault speaks volume's to how important it is to understand what abstractions you're using when you develop an application.  About 6 or so months ago I read two really interesting books to help me understand what abstractions I was building on top of when I wrote a .NET app.  If your interested in whats going on under the hood of your .NET app check out Don Box's Essential .NET, and Shared Source CLI!

Is the "web" going away?

With the advent of "smart clients" I fear that the web as I know it is going away.  As a software engineer I'm not sure how I feel.  Over the last three to four years I've built a lot of browser based applications for my clients.  Over that time I didn't question what I was doing and if it was the best solution.  The web was so cool, how could it be wrong?  Now Microsoft goes and creates all this technology that makes me question everything.  With "no touch deployment", and the application updater block I can develop a rich smart client while keeping many of the advantages that the browser based application provides. 

Now in reality I don't see the web going anywhere.  I believe I will continue to develop browser based applications for many many clients and I think it will continue to provide them with a lot of value.  However, I do believe that smart clients are the future.  I see a world where smart clients provide a richer and more robust interface for users of the applications I develop.  Over the next couple of months I'm going to be working on a content management system that will have a browser based front end, as well as a smart client front end.  Perhaps it will be the beginning of a shift in focus for my development efforts...then again perhaps it won't.....stay tuned.

Am I designing "good" libararies?

I'm sure almost everyone has already come across Brad Abrams posts on designing good libraries but just in case you haven't here are some links:

• Designing Good Libraries Part 1
• Designing Good Libraries Part 2
• Designing Good Libraries Part 3

A while back I wrote a .NET Guidelines document that contained a set of recommendations for designing libraries in .NET, I think its about time to update them! 

What is your favorite browser based HTML WYSIWYG editor?

I'm about to start development on a content management system for several clients and am looking for the absolute best HTML WYSIWYG editor out there.  The one's I've found so far are:

• FreeTextBox
• RichTextBox
• soEditor
• Active Edit
• Html TextBox
• r.a.d. editor

Has anyone had any experiences with any of these?  I'm thinking about turning the CMS I'm going to be developing into a commercial product so am interested in finding the best editor out there...any advice?