June 2006 - Posts
What is sure to be the first in many implementations of Linq over external data sources, Fabrice has put together Linq to Amazon which allows querying Amazon via Linq. I’ve been wanting to experiment with the extensibility points within LINQ to put together a similar sample but haven’t had the time recently. Hopefully in the coming weeks I’ll get back some of my “free time” to do some experimentation.
Update: Fabrice has a second post that touches on the “implementation fore steps” for Linq to Amazon. Learn to love IQueryable! …and iqueryable.com for that matter 
A couple days ago in my “What happens when an agile team falls flat on their face?” post I talked about how last week we failed to deliver on our commitments. In the comments of that post Howard asked “
Doing one week iterations can be tough. They leave very little room for error and place a lot of pressure on the development team to perform well each and every day. Throughout the course of our project we’ve tried a bunch of different things to help ensure we meet our commitments and know when things are going off course. While we’re no-where near perfect I think we do a very good job of identifying what needs to be done within our iterations as well as tracking our progress throughout the week.
At the beginning of each week business puts on a demo for others within our organization as well as a few select clients. This allows everyone to get a feel for what was done the previous week and provide feedback on what they like, and don’t like. Not meeting commitments for an iteration makes the process of giving the demo difficult since it is glaringly obvious to everyone involved when things weren’t quite finished. Fun times!
Following our iteration demo we go into a retrospective where we talk a lot about the previous iteration. We identify “memorable moments” and classify them into several categories (enjoyable, frustrating, puzzling, more of, less of, same of). As a result of this process we’re able to identify areas that we need to take correctionary action, as well as areas where we’ve improved. The retrospective is a vital part of our process as it ensures we don’t fall into a rut and struggle with the same issues over and over. In short it helps us be crazy-agile.
After completing our demo and retrospective we start talking about the stories we have planned for the next week. Par the course, business always has more stories then we can fit into a week. From a development perspective we need to ensure we can complete the stories we commit to. Since everyone is always going to want development to be able to go faster and complete more stories in a shorter time period its important that we don’t commit to too much (or too little). If nothing else we need to be consistent with our estimates and reliably deliver what we commit to.
To determine the number of stories that we should breakdown into tasks we look at our previous iteration velocity (aka “yesterday’s weather”). Using yesterday’s weather provides us with a starting point for our story breakdown. While it’s nowhere near perfect it helps get us started with the approximate number of story points we’ll be able to complete in the coming week.
Once we’ve identified the approximate number of story points we break each individual story into tasks. This has proven to be a critical part of our process. This past week during our retrospective we realized that we did a piss poor job during our task breakdown last week. We had about 15 task cards on the board for last weeks iteration while this week (which is just the completion of our previous weeks work) we have 40+ cards on the board. It goes without saying that we didn’t identify a lot of work that was required for the story.
After breaking down our stories into tasks we estimate each task. Over the past 12 weeks each member of the development team selected cards that they felt they could estimate and did so. To try something different this past week we estimated the tasks as a group using “poker estimation”. Once we have estimates on all of the tasks, as well as cards for all the non programming tasks that happen throughout the week we add everything up to see if everything will fit within the available time. More often than not we have more work than time available so business goes through the difficult process of prioritizing stories and figuring out which they’d like to fit into the iteration.
After the stories are selected by business we place the cards on our large magnetic whiteboard. Story cards go from top to bottom on the left hand side of the board and task cards go from left to right across the board. When we start a task we grab the card off the board and mark it as started with a brown marker and our initials. When a task is completed it is marked with a green marker and the amount of time spent on the task is placed on the front of the card below the original estimate.
By marking our cards with brown and green markers we’re able to see how we’re progressing through the iteration. If by Wednesday afternoon half the board isn’t green we start to determine if we’re going to need to cut tasks or stories from the iteration. As mentioned earlier this is where task breakdown proves to be key. If we’ve done a lousy job of breaking down our tasks we may get a bad read on how we’re progressing. If on the other hand we’ve done a good job of breaking down our tasks then the board gives us a perfect read on where we are with things.
Of course we never have and never will be able to identify every task at the beginning of the week. As new tasks are identified we create new cards with an original estimate of 0 and an Updated Estimate of whatever is appropriate. In previous iterations we would place the new tasks on the board along with the original tasks, however, for our current iteration we’ve decided to track our new task cards by putting a yellow line above the card on the board to ensure we don’t get a false sense of progress. If we’ve completed 25% of the cards on the board but all of those cards were new task cards that could be a sign of trouble.
As we get towards the end of the week we begin to get bug reports from QA. The bugs are placed on index cards and put up on the whiteboard with all the original tasks. The cards are typically separated into a “bugs” section on the whiteboard. “Bug” tasks are handled the same as normal task cards except for when they’re very small in which case they usually get combined into a single card for time tracking purposes.
We follow the above process which does a pretty good job of ensuring we don’t have weeks like last week. When we do have a bad week the process is there to ensure we do the things necessary to prevent it from happening again (hopefully). What do you do to help increase your likelihood for success?
Much has been written about the benefits of doing Agile software development. I won’t spew the various catch phrases and mantra’s that you’ll find across the web on agile, sufficient to say it has proven to work well for many people. With that said, there is a great number of developers who haven’t tried agile software development. Some because they just don’t buy into it, others because their organizations won’t let them, and others because they don’t know what it is. With all that’s been written about the benefits of Agile I think we often times forget to tell people that it isn’t the easiest thing in the world to do. While thats understandable given the fact that many people are trying to bring people to agile, not scare them away, I think we do run the risk of giving people a false sense of what is involved with doing agile development. It can be hard, it can be frustrating, and it can be painful.
This past week was one of those painful times. At the beginning of the week we went through our planning session and identified a single story that we were going to focus on for the week. While the story was quite large it wasn’t something that we didn’t think we could handle. As the week progressed we all felt we were making decent progress (we weren’t) and we all felt that we’d be able to meet our commitments (we didn’t). By Thursday afternoon many of us started to realize that we were in trouble. We’d have to put together a Herculean effort in order to finish everything that was left by Friday afternoon. As you might have guessed we didn’t make it, in fact we didn’t even come close. It sucked. To make matters worse we didn’t cut our loses on Friday afternoon. We all decided that we wanted to try and put some time in over the weekend to complete the story. After-all we had committed to finishing the story and we’ve done a pretty good job of meeting our commitments over the last several months. While we made an attempt it wasn’t nearly enough. It sucked…big time! To make matters worse today we broke down the remaining tasks required to complete the story and realized we had enough remaining work to fill up the entire next week. SH*T! How depressing.
When doing Agile software development your going to fall on your face from time to time, and it’s going to be painful. The Green bars might make you feel a little bit better but its still going to hurt. Lucky for you you’ll get to do it all over again “next week”.
What is sure to be the first in many implementations of Linq over external data sources,
Fabrice has put together Linq to Amazon which allows querying Amazon via Linq. I’ve been wanting to experiment with the extensibility points within LINQ to put together a similar sample but haven’t had the time recently. Hopefully in the coming weeks I’ll get back some of my “free time” to do some experimentation.
Shyam Pather, development lead of the ADO.NET vNext team, has announced two screencasts on ADO.NET Entities. If your interested in seeing what Microsoft has in store for us in the next version of ADO.NET you’ll want to have a look.
Here’s a high-level breakdown of what you’ll see in Part 1:
0:00-2:25 Intro and demo of basic ADO.NET 2.0 code.
2:25-8:00 Creating a conceptual data model
8:00-12:30 Using the Map Provider to query with Entity SQL
12:30-16:05 Explicit relationship navigation in Entity SQL
16:05-17:50 Accessing result metadata via IExtendedDataRecord
17:50-21:55 Polymorphic Queries
21:55-23:40 Filtering on entity type at the server
Part 2 builds on this and covers the following additional topics:
0:00-6:15 Obtaining results as objects
6:15-9:34 Polymorphic queries with results as objects
9:34-11:53 Removing the connection handling code
11:53-14:48 Using LINQ to express queries
14:48-21:02 Adding and updating entities
This morning I came across Fabrice’s post about the ADO.NET Entity Framework documents making a re-appearance on MSDN. A couple weeks ago an initial version of the documents where published on MSDN and then promptly pulled down. I printed the docs out before they were pulled so I had a chance to read them over. For the most part I wasn’t that excited about what they had to say. I should note that I read them very quickly and didn’t give them much attention.
After reading the ADO.NET Entity Framework Overview I’m pretty excited to get my hands on some bits. It sounds like the Entity Framework that Microsoft is developing could be the next great “O/R Mapper”. The Entity Framework is going to come with tooling which will allow developers to map their conceptual domain layer onto their relational database. The conceptual model as well as the mapping between the model and the relational store will be stored in Xml documents that the new mapping objects within ADO.NET will use to translate between the two. Additional tooling will be provided that will allow a set of .NET classes to be generated from the models developed.
To query data using the Entity Framework developers will use a new query syntax for Entities currently dubbed “Entity SQL”. For those familiar with how many of the current O/R Mappers support querying think of Entity SQL as HQL, OPath (or WORM OPath), or NPath. Entity SQL allows you to write queries using a syntax similar to SQL, however, instead of dealing with tables your dealing with your domain model entities.
When I first saw the ADO.NET Entity Framework documents they didn’t talk much about LINQ. I’m not one who is scared by the idea of writing a query that has to join on several tables so the fact that Entity SQL removed that need didn’t get me all the excited. What’s nice is that since the Entity Framework will have Linq to Entities we won’t need to concern ourselves with writing Entity SQL. Linq to Entities will handle all of that for us so we’ll be able to focus on our domain logic and not worry about how we need to formulate our Entity SQL queries for the data we require.
Still on the reading block is the second paper released to MSDN entitled “Next Generation Data Access – Making the conceptual level real”. Unfortunetly I have some work that needs to get done before I read that one. Regardless, the one thing that is clear is that the next 6–18 months is going to bring a lot of very interesting technologies for those of us interested in finding ways for making our business applications have a strong focus on the domain rather then the relational model.
Today Microsoft made available a prototype for Blinq.
Blinq is a tool for generating ASP.NET websites for displaying, creating, and manipulating data based on database schema. Just point Blinq at a SQL database and it will create a website with pages that display sorted and paged data, allow you to update or delete records, create new records, and follow relationships between tables in your database. You don't need to write SQL queries to use Blinq; LINQ will generate optimized queries for you that request just the data you want to show. Blinq uses the May LINQ Community Tech Preview to access data. The code Blinq creates is simple and easy to customize to fit your needs. Everything in the website Blinq creates is meant as a starting point for a website that meets your needs perfectly, so have fun customizing the pages, experimenting with the code, and making it yours!
To create a Blinq site open up a command prompt and navigate to the C:\Program Files\Microsoft ASP.NET\Blinq folder.
C:\Program Files\Microsoft ASP.NET\Blinq>blinq /t:"c:\Development\MyBlinqSite" /database:Northwind /server
local) /vdir:blinq
After generating your Blinq site a browser opens up that allows you to navigate the tables in your database. (Note: If the ASPNET user doesn’t have permissions to your database you’ll need to grant him the necessary access.) Each table is shown in grid form and provides inline editing capabilities. New records are inserted via a secondary page that makes use of a details view. Every table in the database is accessible via a custom DataContext class and all data entry pages make use of a ObjectDataSource control that links back to the automatically generated “business object” that maps to your database table. Blinq creates a partial class that implements IQueryable<T> for each “business object” as well as methods for saving, updating, deleting, and retrieving data.
Overall, I was mostly underwhelmed. Blinq is basically just a simple code generator for creating list and edit pages for your database. The UI was decent but not overly impressive. I wonder what the long term vision/goal is, perhaps just to increase the buzz surrounding LINQ?
Dave Peck has a post about the performance of LINQ over collections. I’m not that surprised with his numbes given the current status of LINQ but I do wonder what the final numbers will be, and what performance targets the LINQ team is chasing.
While the performance numbers are interesting on their own Dave also diggs into some of the details regarding how the C# 3.0 compiler performs it’s LINQ magic. So have a looksy.
- A big congratulations to Sam for his 100th New and Notable post. I’m sure he’ll make fun of me because only my mom reads my blog (shit, she doesn’t even read it does she!?!?!) and I don’t drive any traffic to his blog. Oh well, such is life.
- Alex has an interesting post on Understanding the future of data: Data 2.0
- It seems to be a good time to leave Microsoft, first Scoble and now Bill G (checkout his channel 9 video). I wonder if they’ll fall apart now?
- Project Glidepath – “Project Glidepath is designed to provide the knowledge you, as a MicroISV, need to be successful by providing step-by-step instructions for everything from how to get started with Windows Presentation Foundation to how to write and publish a press release. Please give Project Glidepath a test drive and let us know what you think.”
Wes has a interesting post about XLinq and points to an even more interesting paper by Erik Meijer titled:
XLinq: XML Programming Refactored (The Return of the Monoids) . There has been a lot of hype in the blogosphere surrounding LINQ as well as lots of confusion around the fact that LINQ and DLINQ are not the same thing. For those doing lots of work in Xml XLinq may prove to be the most important of the three.
Another excellent video has been put up on Channel 9 about LINQ. Besides digging into some of the new stuff available in the May CTP of LINQ the video also touches very briefly on the Entity Framework that is coming in the next version of ADO.NET. As I noted in my previous post I really hope Microsoft doesn’t ship two O/R Mapping solutions along with LINQ.
By far the best part of the video is where Anders starts whiteboarding about how LINQ works under the covers. I have a feeling we’re alll going to come to love IQueryable. I agree with Alex and Jonesie that this is a must watch video. Enjoy 
One of the announcements that I’ve heard rumors about coming from TechEd is that Microsoft is going to be releasing two separate O/R Mappers along with LINQ. LINQ to SQL (aka: DLINQ) will be a lightweight O/R Mapper and LINQ for Entities will be more fully fledged O/R Mapper that will be integrated with the Entity Framework that is going to be part of ADO.NET 3 (or 4, 5, 6 depending on how many renames we end up with before it’s released).
One of the biggest problems in the O/R Mapper space is that there are way too many options to choose from. Seemingly everyone has written their own mapper, released it to the public and gone on to attempt to distinguish themselves from the thousands of other mappers available.
Apparently Microsoft has done the same thing with two separate divisions working on their own mapping technology. Rather then merging them into a single solution they’ve decided to make them both available and try and spin them as solving different problems. Yeah right.
Here’s a new idea. Create a single O/R Mapping solution that is integrated with LINQ. Make it extensible so that if it doesn’t fit someone’s specific need they have ways to make it fit. Focus on making it use the Provider model that you’ve made so famous so that we can swap out the behavior if need be. Don’t create two separate mapping solutions. We already have two many, we don’t need too many from you as well.
Bob Martin recently addressed some criticism of Agilists and stated:
Agile is not a silver bullet. Agile is not THE ANSWER. But Agile techniques, and especially TDD, are very powerful techniques to get software written well, and to increase the level of professionalism in our industry.
Agile not a silver bullet? What? You’ve got to be kidding me. You mean it won’t solve all my problems and make my life easy? Unbelievable. I’ve had enough.
One of the things that often gets overlooked when people talk about LINQ is the fact that it is NOT the same thing as DLINQ. DLINQ is an extension of LINQ that turns expression trees into SQL so that LINQ expressions can be used to retrieve information from a SQL Server database. One of the things I’m most excited about with LINQ is that eventually all the major O/R Mappers will support LINQ. So if DLINQ turns out to miss the mark we can still have all the wonderful coolness of LINQ while still enjoying the benefits that our O/R Mapper provide. A couple of brave souls have already started the transition to supporting LINQ in their mappers (NPersist & Genome). It’s only a matter of time before someone bakes support for LINQ into NHibernate, WilsonORMapper, and others.
In my The Evolution of a Query API and Our Query API Explained posts I detailed some of the work that we’ve been doing with our query api. We moved from a loosely typed query that had hard coded string literals to something more strongly typed and LINQ-like.
before:
repository.FindAll(“Name = ‘Steve’ AND Age > 21”);
after:
repository.FindAll(Customer.Columns.Name == “Steve” && Customer.Columns.Age > 21);
We’ve also begun work on a CriteriaFormatter class that handles all the formatting of the criteria object into SQL. The basic idea behind our CriteriaFormatter is that a custom ICriteriaFormatter could be written for whatever your O/R Mapper uses for querying objects: HQL, OPath, NPath.
In addition to being able to query our database using our criteria objects I also thought it would be nice to have support for querying in memory objects within a generic List<T>. The generic List class in .NET 2.0 has several very nice methods available on it that allow you to perform operations on the items in the list. The two methods that I was most interested in was .Find(Predicate<T> predicate) and .FindAll(Predicate<T> predicate). Find returns the first item in the list matching your predicate and FindAll returns a list with all the items in the list that match the given predicate.
The question became..how do we turn our Criteria objects into Predicates? After a little bit of research I started experimenting with Lightweight code generation and implicit operator overloads. Our PropertyCriteria objects, which is what results from the Customer.Columns.Name == “Steve” code, was a pretty good place to start.
In order for our criteria objects to turn into predicates I needed to figure what IL was emitted when the following operators were used: ==, >, >=, <, <=, != within C#. With the help of Reflector I was able to figure out how each operator is emitted in IL. With that knowledge in hand I was able to write an AsPredicate method on our criteria objects that used lightweight code generation to dynamically construct a Predicate<T> from our criteria object. The only other things that was needed was an implicit operator overload to turn our Criteria<T> into a Predicate<T>.
public static implicit operator Predicate<T>(Criteria<T> criteria) {
return criteria.AsPredicate();
}
As a result we can now do:
List<Customer> customers = List<Customer>();
// load up with customers
List<Customer> matches = customers.FindAll(Customer.Columns.Name == "Steve");
matches = customers.FindAll(Customer.Columns.Name == "Steve" && Customer.Columns.Age > 27);
matches = customers.FindAll(Customer.Columns.Name.StartsWith("S") | Customer.Columns.LastName.StartsWith("E"));
And I can of course use the same criterias against our repositories that hit our database:
List<Customer> matches = customerRepository.FindAll(Customer.Columns.Name == "Steve");
matches = customerRepository.FindAll(Customer.Columns.Name == "Steve" && Customer.Columns.Age > 27);
matches = customerRepository.FindAll(Customer.Columns.Name.StartsWith("S") | Customer.Columns.LastName.StartsWith("E"));
Note: As you may have noticed I still haven’t figured out how to get a || to work so I’m stuck with a single pipe for “or”. Anyone have any ideas?
Justin Palmer and the crew over at Jaded Pixel have gone live with Shopify. The goal of Shopify is “to making e-commerce easy.” While I haven’t evaluated Shopify in much depth from what I have seen it provides:
- a very nice UI interface for managing an online store
- a robust theming engine
- paypal & authorize.net payment options
- and a seemingly reasonable price structure
So if your looking for a hosted online store solution give Shopify a look.
Note: Those of you with a good memory might remember that Justin did some design work for me on ActiveType a while back. It’s based on this experience that I recommend giving his latest endeavour a look.
In my The Evolution of a Query API post I outlined the details of our query API. In the end we ended up with the following:
List<Customer> customers = repository.FindAll(Customer.Columns.Age == 20 & Customer.Columns.Name == “foo”);
As I mentioned in our previous post the Customer.Columns “collection” is code generated off our database. The Customer class has a static Columns class that contains static properties for each of the columns defined in the Customer table.
public class Customer {
// class definition
public static class Columns {
public static StringColumn<Customer> Name = new StringColumn<Customer>(“Name”);
public static Column<Customer, int> Age = new Column<Customer, int>(“Age”);
}
}
We have a number of different Column types for different data types (StringColumn, DateColumn, etc.) that have custom methods on them that result in the creation of our Criteria objects. On our base column class we have the following static methods:
- Criteria EqualTo(T value);
- Criteria NotEqual(T value);
- Criteria LessThan(T value);
- Criteria LessThanOrEqual(T value);
- Criteria GreaterThan(T value);
- Criteria GreaterThanOrEqual(T value);
These utility methods allow us to do things like:
repository.FindAll(Customer.Columns.Name.EqualTo(“Steve”));
Our column classes also have operator overloads to provide the nice syntax that is shown in the initial sample. As you can imagine each of our operator overloads simply delegates to one of the above methods.
public static Criteria operator ==(T value) {
return EqualTo(value);
}
In order to chain our various criteria expressions together we implemented an operator overload on our base Criteria class for the & and | operators.
public static Criteria operator&(Criteria lhs, Criteria rhs) {
return new AndCriteria(lhs, rhs);
}
The final piece of the puzzle is writing a parser that can then analyze a criteria object and convert it to the correct string representation. In our case we’re usually converting it to SQL which our criteria class handles. Ideally when our evolution is a little further along we’ll have a separate class that handles the parsing and creation of the SQL so that we can then write other parsers for whatever OR Mapper we ultimately decide on.
public string ToSqlWhere(Criteria criteria) {
return “WHERE “ + GetColumnMapping(criteria.PropertyName) + “ “ + GetOperatorExpression(criteria) + “ “ + FormatValue(criteria.Value);
}
The above is a extremely simple contrived example that shows how one might write a “parser” that could handle converting a Criteria object to SQL. Obviously any real parser would have to have a lot more logic to handle AndCriteria’s, OrCriteria, and all the other conditions that are supported.
As I was chatting with Jeff Gonzalez tonight about how I implemented our query api he pointed me to this demo video of “The Last Component”. It appears that they’ve stolen every one of my ideas and then some. Since they’re actually selling their framework it’s likely a lot more polished then what I have so have a look at their demo and grab their latest version to give it a spin.
As our query api continues to evolve I’ll post some additional details about its progress along with some more sample code for the various peices in the puzzle (Columns, Criterias, Parsers, and etc.). As always let me know what your interested in hearing more about.