.NET
A while ago, amid my fascination with Rails, I did some investigation of the migrations support that Rails provides. To this day my Ruby on Rails Migrations Explained post is by far the most visited page on my site. I always find it interesting that despite my blog being almost entirely on .NET related topics, one of the few Ruby on Rails posts I've written is the most popular.
Anyway, when I originally investigated Rails I thought it was a great solution for handling the evolution of a database. In fact, I believed in it enough to do some prototype work with it at work to try and get it to support everything we needed it to for the databases that we support (Oracle + SQL Server). In the end we ended up not going with Rails for a variety of reasons. To this day I still regret the decision as we're hand rolling scripts for each of our databases and not especially enjoying it. All of this is why I was very interested to come across this post on Database migrations for .NET. Unfortunately it doesn't yet support Oracle, and probably would need some tweaking to work in our environment. Regardless, I'll certainly be having a look at it as a potential solution to the problems that we face for evolving our database schema.
Database migrations for .NET is provided as part of the Castle Project Generator project. You can find more information about it on the Generator Wiki page.
Today I tracked down an interesting bug within a set of code in our Query API. The code is responsible for converting a Criteria<T> object to a Predicate object. As I mentioned in several of my previous posts, our Query API allows us to create strongly typed queries to access our database. For example, to query all customers who live in London we can issue the following query:
List<Customer> customers =
Customer.FindAll(Customer.Columns.Country == "London");
We can also use the same query (or criteria), to query in memory collections that support searching by a predicate, such as List<T>. To query an in memory List<T>, we can use the following code:
List<Customer> customers = // get list of customers from somewhere
List<Customer> londonCustomers =
customers.FindAll(Customer.Columns.Country == "London");
In order to support this functionality, we have some code that creates a Predicate<T> from a PropertyCriteria<T>. The code uses the lightweight code generation features in .NET 2.0 (DynamicMethod, ILGenerator) to generate the method on the fly by spitting out the appropriate IL OpCodes. What we came across today was that the current code did not consistently work when querying by date. So if we issued a query such as:
Customer.FindAll(Customer.Columns.HireDate.HappensAnytimeDuring(dateOfInterest));
We ended up with completely random results. After spending a little bit of time looking at the code today at work, I shelved the issue in favor of more important needs. Out of curiosity I couldn't help but return to investigate the issue this evening. After running a couple tests, and reconfirming that the results were completely random, I turned to Reflector to see if I could uncover anything. After a little bit of digging I would up in the op_LessThan method of System.DateTime. I then switched over to System.Int32 (since int's where working swimmingly) and realized there wasn't a corresponding op_LessThan method. Of course this is because an Int32 is a primitive, while a DateTime is not. Well to make a long story short, I was using a set of OpCodes that (as far as I can tell) is designed for comparing primitives, namely: OpCodes.Ceq, OpCodes.Clt, and OpCodes.Cgt. Switching my predicate generation code to call the Equals, op_Inequality, op_LessThan, opGreaterThan, op_LessThanOrEqual, and op_GreaterThanOrEqual methods instead of using the OpCodes mentioned above resulted in consistent, and more importantly, correct results.
Once again via Sam via Mark Miller
Here's a couple options provided by Mark:
There are two ways to share templates. The first is to place the
templates you want to share inside a folder category (on the template editor options page), then right-clicking this folder and selecting "Export
Folder". Then you can distribute this template folder to other team members who can import it by right-clicking on the category list on the template editor options page and selecting "Import Templates.."
The second is to check your template files directly into source control,
with one team member having read/write access and the other team members having read-only access. Currently, all templates are stored into a single binary file (for a fast read), and a backup xml file. The binary files are read when templates are needed (if a binary file isn't found then the XML file is read). Both files are written when you save template settings in the options dialog. So I would probably suggest checking in/out only the binary files since they are about one seventh the size of the XML files. Templates are stored in the "Editor\Templates" folder inside the Settings directory.
Technorati tags:
coderush
via Sam via Mark Miller
As I mentioned in a previous post we recently purchased a couple CodeRush licenses. As such, we're trying to figure out how to use it. This post is meant to provide details about how you can import/export CodeRush settings between team members.
CodeRush settings are stored here:
{SystemDrive}\Documents and Settings\{CurrentUser}\Application Data\CodeRush for VS .NET\{Version}\Settings\
You can import/export these by simply performing an XCopy. Setting
sub-folders match the folder organization in the Options dialog. You don't have to export the settings in the Loader folder (used for load on demand), since those are maintained automatically by the products.
It would be nice if you could change where it stores settings so at least some could automatically be shared via source control. For the time being this should do the trick.
Technorati tags:
coderush
We recently purchased a couple CodeRush licenses and I'm looking for the answers to a couple questions.
- How do you import/export CodeRush settings?
- How do you share templates?
- Who has a Template & Keyboard Cheat Sheet (something like this ReSharper one)
- Are there any killer DXCore plugins that I should check out?
Technorati tags:
coderush,
vsnet
Alex has been playing with IronPython lately and it seems he's been hooked.
His posts have certainly peeked my interest. Hopefully I'll get some time to play around, but in the meantime I'll continue to checkout Alex's adventures!
tags: IronPython
Sam has a couple posts outlining some of the great fun we had this past week.
I’ll leave all the gory details to Sam. However, I would recommend you take a look at the Exception Handling and Logging Blocks within Enterprise Library (and all the other blocks for that matter) if you haven’t done so recently. Before this past week it had been a while since I looked at EntLib. After spending a day working with the Logging and Exception Handling block along with a couple hours of investigation earlier in the week I have to say it is a very nice framework for ensuring your applications are properly instrumented and all your exceptions are handled appropriately (based on policy).
A couple months ago I posted about our query API along with an explanation of how it works. Our query API allows us to express our queries using strongly typed C# syntax:
List<Customer> customers = repository.FindAll(Customer.Columns.Age == 20 & Customer.Columns.Name == “foo”);
One of the things I pointed out in my previous posts was that I couldn’t overload the && and || operators directly since the framework doesn’t allow such craziness…at least not directly.
In particular, it is not possible to overload member access, method invocation, or the =, &&, ||, ?:, checked, unchecked, new, typeof, as, and is operators.
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/csspec/html/vclrfcsharpspec_7_2_2.asp
Over the last month I’ve done a little investigation on the topic to see if and how I can get && and || to behave the way I want. This evening I came across the Conditional logical operators page on MSDN which gave me the answer I was looking for:
- The operation
x && y is evaluated as T.false(x) ? x : T.&(x, y), where T.false(x) is an invocation of the operator false declared in T, and T.&(x, y) is an invocation of the selected operator &. In other words, x is first evaluated and operator false is invoked on the result to determine if x is definitely false. Then, if x is definitely false, the result of the operation is the value previously computed for x. Otherwise, y is evaluated, and the selected operator & is invoked on the value previously computed for x and the value computed for y to produce the result of the operation.
- The operation
x || y is evaluated as T.true(x) ? x : T.|(x, y), where T.true(x) is an invocation of the operator true declared in T, and T.|(x, y) is an invocation of the selected operator |. In other words, x is first evaluated and operator true is invoked on the result to determine if x is definitely true. Then, if x is definitely true, the result of the operation is the value previously computed for x. Otherwise, y is evaluated, and the selected operator | is invoked on the value previously computed for x and the value computed for y to produce the result of the operation.
Since we already have the & and | operators in place it was simply a matter of overloading the true and false operators to both return false. This results in the & and | operators always being called which in turn results in the two criteria objects being turned into an AndCriteria/OrCriteria!
So now we can express our criteria expressions using the && and || syntax we’re accustomed to.
repository.FindAll(Customer.Columns.Age == 20 && Customer.Columns.Name == “foo”);
repository.FindAll(Customer.Columns.FirstName == “Foo” || Customer.Columns.LastName == “Bar”);
The relevant operator overloads are shown below.
public static bool operator true(Criteria<T> criteria) {
return false;
}
public static bool operator false(Criteria<T> criteria) {
return false;
}
public static Criteria<T> operator &(Criteria<T> lhs, Criteria<T> rhs) {
return new AndCriteria<T>(lhs, rhs);
}
public static Criteria<T> operator |(Criteria<T> lhs, Criteria<T> rhs) {
return new OrCriteria<T>(lhs, rhs);
}
In my The Evolution of a Query API post I outlined the details of our query API. In the end we ended up with the following:
List<Customer> customers = repository.FindAll(Customer.Columns.Age == 20 & Customer.Columns.Name == “foo”);
As I mentioned in our previous post the Customer.Columns “collection” is code generated off our database. The Customer class has a static Columns class that contains static properties for each of the columns defined in the Customer table.
public class Customer {
// class definition
public static class Columns {
public static StringColumn<Customer> Name = new StringColumn<Customer>(“Name”);
public static Column<Customer, int> Age = new Column<Customer, int>(“Age”);
}
}
We have a number of different Column types for different data types (StringColumn, DateColumn, etc.) that have custom methods on them that result in the creation of our Criteria objects. On our base column class we have the following static methods:
- Criteria EqualTo(T value);
- Criteria NotEqual(T value);
- Criteria LessThan(T value);
- Criteria LessThanOrEqual(T value);
- Criteria GreaterThan(T value);
- Criteria GreaterThanOrEqual(T value);
These utility methods allow us to do things like:
repository.FindAll(Customer.Columns.Name.EqualTo(“Steve”));
Our column classes also have operator overloads to provide the nice syntax that is shown in the initial sample. As you can imagine each of our operator overloads simply delegates to one of the above methods.
public static Criteria operator ==(T value) {
return EqualTo(value);
}
In order to chain our various criteria expressions together we implemented an operator overload on our base Criteria class for the & and | operators.
public static Criteria operator&(Criteria lhs, Criteria rhs) {
return new AndCriteria(lhs, rhs);
}
The final piece of the puzzle is writing a parser that can then analyze a criteria object and convert it to the correct string representation. In our case we’re usually converting it to SQL which our criteria class handles. Ideally when our evolution is a little further along we’ll have a separate class that handles the parsing and creation of the SQL so that we can then write other parsers for whatever OR Mapper we ultimately decide on.
public string ToSqlWhere(Criteria criteria) {
return “WHERE “ + GetColumnMapping(criteria.PropertyName) + “ “ + GetOperatorExpression(criteria) + “ “ + FormatValue(criteria.Value);
}
The above is a extremely simple contrived example that shows how one might write a “parser” that could handle converting a Criteria object to SQL. Obviously any real parser would have to have a lot more logic to handle AndCriteria’s, OrCriteria, and all the other conditions that are supported.
As I was chatting with Jeff Gonzalez tonight about how I implemented our query api he pointed me to this demo video of “The Last Component”. It appears that they’ve stolen every one of my ideas and then some. 
Since they’re actually selling their framework it’s likely a lot more polished then what I have so have a look at their demo and grab their latest version to give it a spin.
As our query api continues to evolve I’ll post some additional details about its progress along with some more sample code for the various peices in the puzzle (Columns, Criterias, Parsers, and etc.). As always let me know what your interested in hearing more about.
Have I mentioned that I like LINQ
When developing applications that use a relational database we often spend a reasonable amount of time constructing queries for retrieving the data that we need out of the database. The way by which we (as developers) go about retrieving this data varies considerably across applications. Some people build Sql Command objects and add their parameters to it, others build a query object for their O/R Mapper and pass it on through, and still others build the SQL for retrieving their objects from scratch from within their C# code.
Over the course of the last several months we’ve changed the way we query for data a number of times. In the beginning we started with hand coding expressions in string literals like so:
List<Customer> customers = repository.FindAll(“Age > 20 AND Name = ‘foo’”);
While this did the job it wasn’t ideal since if a column name changed it wouldn’t be caught at compile time. We also ended up with some logic for dealing with certain cases spread across our code base. To help reduce the duplication of code that we started having we created “criteria” objects to encapsulate that logic. Our code became something like:
Criteria c = new AndCriteria(
new PropertyCriteria(Customer.Columns.Age, 20, Operator.GreaterThan),
new PropertyCriteria(Customer.Columns.Name, “bar”)
);
List<Customer> customers = repository.FindAll(c);
This helped reduce the amount of code that was scattered across our code base and got rid of the problems that having column names within string literals introduced, however, our query became pretty verbose. The next step in the evolution of our query API was to take the best of the first approach and combine it with the best of the second approach. In the end we ended up with this:
List<Customer> customers = repository.FindAll(Customer.Columns.Age == 20 & Customer.Columns.Name == “foo”);
Now we have the nice compact syntax of the original query, nice strongly “typed” columns (via code-gen), and the benefits of having all the custom logic that happens behind the scenes encapsulated in our base criteria class (which is what gets returned as a result of the expression above). The other benefit of this approach is that it isn’t tied to any particular O/R Mapper or data access strategy. If we decide we want to move from one O/R Mapper to the other its just a matter of writing a single criteria “parser” that can translate our criteria objects into OPath, HQL, OQL, or etc. The next step in our evolution will be getting the same criteria expression to also work with the various find operations we have on in memory collections such as List<T>.FindAll().
Scott Bellware has an excellent post entitled “Mort or Elvis: A Question for a Bygone Era” which is right on the money. If we’re going to make the software development profession a better place we need to take a fresh look at how we’re building tools and who we’re targeting when we build them. We can’t stand still and assume our assumptions of yester-year are still valid, especially if they weren’t ever valid. I’m not sure if or when JetBrains is releasing their standalone IDE for .NET but I have a feeling if it ever does come about there will be a lot of .NET developers jumping off the sinking VS.NET ship.
I’ve recently been experimenting with making some changes to our domain layer to help simplify our design and reduce the number of useless objects that were cropping up throughout our code base. In our previous model we had “Manager” classes that, well, managed stuff. Our managers closely followed the Repository pattern as defined in Domain Driven Design.
public class CustomerManager {
ObjectManager objectManager = new ObjectManager();
public SaveObjectResponse SaveCustomer(Customer customer) {
return objectManager.Save(customer);
}
public bool DeleteCustomer(Customer customer) {
return objectManager.Delete(customer);
}
public List<Customer> FindAllCustomers() {
return objectManager.FindAll<Customer>();
}
// etc…
}
What this resulted in was a whole bunch of classes that looked a lot like the above. I’ve used the ActiveRecord pattern a fair amount in the past so in an attempt to simplify our design we ended up adding some methods directly on top of our domain objects so that all of the above code disappeared. To make them dissappear we added a base entity that defines a number of methods for saving, deleting, and finding domain objects.
public abstract class BaseEntity<T> {
static IDatabase<T> db = new Database<T>();
public SaveObjectRespnse Save() {
db.Save(this);
}
public bool Delete() {
db.Delete(this);
}
public static List<T> FindAll() {
db.FindAll();
}
}
public class Customer : BaseEntity<Customer> {}
While this cleaned up our code and made it so we could get rid of a number of objects it has caused one issue. We now have statics, which means we have a testability issue. In order for us to enable things to be tested we needed a way to inject a IDatabase<T> into our Customer class. Since our base entity has static methods that require the database it requires our database instance to be static. This prevents us from using dependency injection to inject our mock database for testing. As a short term solution I added a means for registering and unregistering a database like so:
public static void RegisterDatabase(IDatabase<T> dbToRegister) {
db = dbToRegister;
}
public static void UnRegisterDatabase(IDatabase<T> dbToUnregister) {
if(db == dbToUnregister) {
db = defaultDb;
}
}
While this works its far from ideal.
[Test]
public void DoSomethingThatDoesNotNeedTheRealDatabase() {
MockDatabase<Customer> mockCustomerDb = new MockDatabase<Customer>();
mockCustomerDb.FindAllResult = myDummyCustomerList;
Customer.RegisterDatabase(mockCustomerDb);
try {
// do some stuff that calls Customer.FindAll();
}
finally {
Customer.UnRegisterDatabase(mockCustomerDb);
}
}
As a result of having some ugliness due to the use of statics I think we’ll likely move to having a “Manager” like base class that supports generics. This will have the same advantages of the current implementation and cause less issues when it comes to testing.
public class Repository<T> {
public SaveObjectResponse Save(T item) {…}
public bool Delete(T item) {…}
public List<T> FindAll() {…}
}
I wonder how the folks who are using Castle’s ActiveRecord handle this issue. From browsing through their code base it doesn’t appear that they make it so you can mock out the ActiveRecord’s behavior easily which typically results in a lot more tests hitting the database then necessary.
Harry Pierson has an interesting post entitled “The Dual Schema Problem” where he talks about the problems that we face in mapping our object model to our relational model. Although todays mappers help reduce the amount of work required to get data out of our relational model and into our object model they don’t help make our lives easier when it comes to managing our schema. We still have to maintain our schema in two places. And we still (for the most part) have to start with our database schema and work our way back to our objects.
In an ideal environment the “schema” of our application would reside in the same place our object model lives. As our object model evolved so to would our schema. We’d work in a single “environment”, we wouldn’t jump between VS.NET and SQL Server, and we’d be more productive.
Rather then posting a short novel on DLinq I decided to cut my previous post “short” and include some more highlights in this follow up post:
- Take(5) – SELECT TOP 5
- Skip(10) — Skips the first 10 records
- Supports “including” related data to prevent additional roundtrips to the database via .Including(c => c.Orders) syntax
- Supports string functions
- StartsWith => ColumnName LIKE ‘foo%’
- EndsWith => ColumnName LIKE ‘%foo’
- Contains => ColumnName LIKE ‘%foo%’
- Length => LEN(ColumnName) > 30
- Substring => SUBSTRING(ColumnName, 0, 5)
- ToUpper => UPPER(ColumnName)
- ToLower => LOWER(ColumnName)
- Concurrency support – OptimisticConcurrencyException is thrown when two users try to modify the same object.
- Supports for direct SQL:
var products = db.ExecuteQuery<Product>(
"SELECT [Product List].ProductID, [Product List].ProductName " +
"FROM Products AS [Product List] " +
"WHERE [Product List].Discontinued = 0 " +
"ORDER BY [Product List].ProductName;"
);