January 2009 - Posts

First looks at the Live Framework SDK

I took some time to create the MyPhotos sample for the Live Framework SDK (currently in CTP).

I’m impressed with the consistency of the programming model.  You can get quite a bit done using just a few objects: A LiveOperatingEnvironment, Mesh, MeshObject, and DataFeed.

In only a few pages of code, I’ve got a desktop application that allows me to create folders in the mesh, upload pictures, and search for them.

the code looks amazingly familiar to any C# developer (I wrote is C#. I believe any VB.NET developer would say the same about the VB.NET version).

There are a few items to point out in the API.  this method shows how you can create a query to find data feeds in a given mesh object. It returns any mesh Files in a single mesh object:

   1: DataFeed GetFileDataFeed(MeshObject meshObject)
   2: {
   3:     DataFeed theDataFeed = (from dataFeed in meshObject.CreateQuery<DataFeed>()
   4:                             where dataFeed.Resource.Title == LIVE_MESH_FILES
   5:                             select dataFeed).FirstOrDefault<DataFeed>();
   6:  
   7:     return theDataFeed;
   8: }

It’s as simple as running a query against the mesh object.  more importantly, the CreateQuery<T>() method returns a LiveQuery<T> object, and LiveQuery<T> implements IQueryProvider!  All the query execution will happen at the server. The live framework parses the query and minimizes the traffic by pushing logic to the cloud rather than pulling data to query from the cloud.

As you explore the rest of the Live APIs you’ll find that many of the APIs that search for data in the cloud make use of IQueryProvider.

Creating objects is as simple as adding them to the correct collection.  Also, you can add other properties to the resource collection associated with an object. It’s four lines of code to upload and annotate a picture with a rating:

 

   1: DataEntry dataEntry = theDataFeed.DataEntries.Add(filestream, filename);
   2:  
   3: // Add user data (ratings) to data entry
   4: dataEntry.Resource.SetUserData<Ratings>(theRating);
   5:  
   6: // Required to see user data on all devices/Live Desktop/Data Model Browser
   7: dataEntry.Update();
   8:  
   9: // Make sure everyone is synchronized
  10: theDataFeed.SyncEntries.Synchronize();
  11: theDataFeed.Update();

More coming soon… There’s cloud applications in my future.

MSDN Developer Conference -- Detroit: January 22

There is still time to register for the MSDN Developer Conference in Detroit, this coming Thursday.

If you missed PDC, this is a chance to get a look at the major content announcements from that conference.  You’ll get sessions on the Azure Services Platform for Cloud Computing, Client and Presentation Technologies, and Tools, Languages and Platforms.  You’ll learn how to build applications for Azure the Live Platform, and Live Mesh Services, and how to use SQL Data Services for storage in the cloud. You’ll see what’s next in ASP.NET, WPF, and Silverlight. You’ll learn what’s next for C# and VB.NET, what is Oslo, what’s the F# buzz about, and what’s coming in VSTS 2010.

That’s in addition to side events like the Community Courtyard, and Women Build.

I’m giving the talk on the future of C# and Visual Basic, and I’m thrilled with the content.  You can register here: http://www.msdndevcon.com/Pages/Detroit.aspx

Entity Framework and LINQ to SQL

1/16:  Updated with corrections from Readers.

I’d hoped to completely avoid any discussion surrounding the future (or futures) of Entity Framework and Linq2Sql.  And yet, my LINQ talks at CodeMash got hijacked with questions about these two libraries.

I tried to sidestep, but there was no hope. Stephen Forte already proclaimed LINQ to SQL dead. What more could be said?

Well, I do hope it’s not as simple as that sound bite, so here are my thoughts.

There’s no doubt that Microsoft will have one data access strategy. There’s also no doubt that the name for that single data access strategy is Entity Framework. (Source)

However, I don’t believe that means concepts, designs, and code from LINQ to SQL won’t be part of that overall strategy.

I prefer Linq 2 SQL now

Entity framework supports a high level of abstraction. LINQ to SQL supports a wide breadth of abstractions. That, in my mind, makes LINQ to SQL a better design.

For example, consider lazy loading. Many people have noted that Entity Framework does not enable developers to specify lazy vs. eager loading at this time. L2S does: You can specify that certain relationships should always be followed for an entire application. (See DataContext.LoadOptions, and DataLoadOptions.LoadWith, or DataLoadOptions.AssociateWith).  You can also change these properties at runtime, so you can specify eager or lazy loading on a query – by – query basis.

EF does support lazy and eager loading now:  Its in the .Include("EntityName") on the query. Thanks Diane.

Another example is the DataContext.ObjectTrackingEnabled property. As you can logically guess, there is some overhead associated with tracking object IDs to correctly track changes on objects back to the changes in the database. You don’t want to pay that penalty if a particular application is only querying data, not changing it.  Set ObjectTrackingEnabled to false, and watch your application speed up. (As always, measure changes meant to improve performance, or you are not doing engineering).

L2S also supports ways to execute dynamic SQL, or execute stored procs. I don’t start using either of those immediately. However, it’s valuable to know that there are options when the default behavior doesn’t meet expectations in a particular scenario.

Update:  Entity Framework does support Stored Procs and Entity SQL, just not T-SQL (Also from Diane)

And all those points above are examples of the larger design decision:  L2S supports a high-level abstraction, but provides ways to use a lower level abstraction when your application requires.

The Evil Enterprise

Entity Framework is an enterprise level framework that was specifically designed to work with large, existing (some may say legacy) databases. Personally, I believe that every scenario being optimized by the EF team begins, “So you have this database…”. 

Too many of our customers are in the startup, or emerging company phase. Greenfield. No existing storage model. Furthermore, our projects are lean projects. The storage schema (whether database or file based) will be changing every iteration.  At least until some release milestone is reached. Entity Framework doesn’t make those scenarios productive yet. They are possible, but they are not productive. These issues are worsened when multiple developers are accessing entity framework model source at the same time.

My opinion, but it seems like EF is targeting developer groups that work more in silos: “The Database Developer”, “the middle tier developers”, “UI Design teams”. We don’t work that way: Lean teams assign work to cross function and deliver slices of value. In one iteration, developers will modify the database, create business objects, and tie those to extensions at the UI layer. EF seems to have been designed around teams that have horizontal roles.

Diane says that EF can regenerate the entire schema, and that's true. My issue is that the manner in which EF writes a new model is very painful for tools like source control, and diff management. make that painful.  If you're currently using a source control system that supports concurrent checkout, the EF regen model doesn't work. 

Do I use LINQ to SQL in new projects?

I do. I know I’m taking some risk, but here’s my analysis, and my reasoning:

First and foremost, LINQ to SQL suits my needs better than EF v1. That’s the most important. Any project I’m starting now will certainly be modified, changed, and updated in the future. I’m willing to buy a future migration for the increased productivity now.

Second, I don’t believe LINQ 2 SQL is a dead end. I believe it will be an onramp to a future (IMHO better) version of EF. This is purely my opinion, and I watch for signs from the EF team, but I don’t believe the functionality in LINQ to SQL that is not in EF will be unceremoniously dropped. I do believe the EF team will have some migration strategy, and will incorporate those new features in EF in some meaningful way.

Third, I believe I have plenty of time. Microsoft is not going to stop including LINQ to SQL in the .NET framework until they are convinced EF is a good migration target for all the current LINQ to SQL users. That day has not arrived. Therefore, I believe I have plenty of time to be productive with LINQ to SQL before I must make a change. I’ll keep looking at EF with each new release. When I see reason to move my existing LINQ to SQL investment to EF, I will do so.

Fourth, I have options.  If I find some future where LINQ to SQL ceases to be supported, and I’m still not happy with EF, I would consider Castle ActiveRecord, NHibernate by itself, or some other technology. If for no other reason than these options, Microsoft will not abandon LINQ to SQL without a migration strategy that make Entity Framework the obvious choice for those customers already using LINQ to SQL.

Fifth, in most of our applications, the data access layer is smaller than many other portions of the application. And, we try to keep it from leaking throughout the application. It’s a non-zero cost, but we try to keep the cost of changing the data access layer small.

Time for a big finish now

Entity Framework is a v1 product. LINQ to SQL is a v1 product. They targeted very different work patterns. Both those can be supported by some future version of Entity Framework. Until then, LINQ to SQL suits my needs better.  That’s what I’ll use until the road map is clearer. I’m sure I’ll be able to leverage that investment in one form or another. I’m not on a dead end.  I’m on a parallel path that will merge onto some future platform.

Posted: Code and Slides from CodeMash

I’ve just uploaded my code and slides from both talks at CodeMash.

I was somehow lucky enough to have two talks at CodeMash:

Extending and modeling the type system using Extension Methods.

Slides

CodeSamples

Understanding Query Comprehensions

Slides

CodeSamples

 

The talk on understanding Query Comprehensions was recorded.  When the CoeeMash tribe posts the recording, I’ll post a link to that as well.

Notes from Poppendieck “Value Stream Mapping”

Wow.  I went to Mary Poppendieck’s CodeMash Precompiler talk on Value Stream Mapping. That was great. It’s rare that I take 5 pages of notes during one conference session. To be fair, the precompiler sessions are three hours, but still, that’s much more value than I usually get out of a conference session.

I’ve condensed, but here are my notes: 

Value Stream Mapping comes from the Lean Business concepts. In most businesses, company value is measured using a balance sheet. Lean businesses think more about cash flow than balance sheet. That’s a major shift in thinking, and affects how you see everything.

Danger: major simplifications coming. I don’t want every MBA correcting my accounting analagy.

For example, your personal balance sheet is probably quite a bit better than your cash flow statement.  Your personal balance sheet will include assets like your home, cars, the future value of your education (which allows you to make more money), cash value of investments, and so on. Of course, it will also include the liabilities against any of those assets: mortgage liability, car loans, student loans, etc.

However, as you go through your daily life, you’re much more likely to think about cash flow: do you have enough cash to make the payments on those liabilities, and pay for other items you want to buy.

The problem (for you as an individual) is that many of the assets on your balance sheet are not fungible. For example, you cannot immediately exchange your home for cash. You can’t use your house to buy groceries.

The analogy works for software as well: partially done work has value (it’s more than not started software), but that value cannot be extracted until the value works its way through the stream to a customer that desires that value. Any value created but not unleashed represents waste.

Mary defined waste as “anything that depletes resources without adding customer value”. From a cash flow perspective, time wasted is a big problem. The longer it takes for value to move from concept to cash, the worse your organization is.

Look at that definition again. “Value” must be seen through the eyes of the customer. It’s not enough for you, internally, to see value. Customers must see the value, because it is customers that will pay for that value.

From this basis, Mary went on to discuss some of the most common forms of waste in software projects.

The biggest single waste is creating code that doesn’t get used.

According to a Standish Group study on custom software, 64% of specified features are rarely or never used. Only 20% are always or often used.

Therefore, the biggest possible savings in software projects is to write less code. However, we have to not write the code that doesn’t add value.

I asked how those statistics can be applied to product development. Mary explained that extra requests in product development usually comes from Lazy Marketing: people request features for dubious reasons: competitors have it, it looks cool, we need to stack up. Instead, marketing should be asked how much revenue those new features would actually produce. To be fair, asking marketing to predict revenue generated from future features is a difficult as asking developers to predict the cost of those features. But it is critically important to try. To address these conditions, developers need to be actively engaged in marketing. And, they need to push for more frequent deliveries of software. Delivery will change a customers perception of what’s important. More frequent delivery will more accurate identification of the most important issues.

The second most important waste is Work in Progress

  • Features that have been requested, but not spec’ed out
  • Features spec’ed, but not coded
  • Features coded, but not tested
  • Features tested, but not delivered.

Look at those four wastes noted above. All four of those items represent queues. Queues represent waste. They containing work in progress; items that hold value that cannot be delivered. More importantly, if the early activities can proceed faster than the later items, you quickly get into a state where your real expected delivery date for new features is “never”.  If you haven’t studied queuing theory, take a brief look here, and ponder it the next time you’re in a traffic jam.

From this, a Value Stream Map describes the flow of activities that starts with a customer need and ends when that need is satisfied.

Look at the emphasis there: It starts with the customer need and it ends when the customer need is satisfied. 

As developers, we too often sub-optimize our portion of the cycle: from feature request to build artifacts, or something similar. When you work with value stream maps, do not do that. You must work end to end in order to identify all the waste. If you don’t know how long a particular activity takes, you should make your best guess. As you present your results, others in the organization will correct any false assumptions and you’ll have a better map. However, if you don’t add some guess, those parts of the process will remain absent, and you won’t see any waste there. that means you won’t fix it. Later, as Mary reviewed our value stream maps, I found it very interesting that she spent much time on processes at either end of what was drawn. Sure enough, activities occurred before we thought the work started; other activities occurred after we thought we were done. Those activities also had waste in them.

After you map the activities, you assign average times to each activity, and each waiting state. The goal is to minimize the wasted time as a request rolls through the system (queues again). Some of the examples should significant opportunities for improvement: activities that had 3 weeks of useful value-added tasks actually took more than a year. That sounds silly, and yet, the map of the activities (without the times shown) looked completely reasonable. The act of writing down the map pointed to incredible waste that could be easily attacked, with incredible benefit.

I’m assuming that most of you reading this blog are engaged in software development activities. That means you may want to create three different value stream maps: one for product releases, one for feature requests, and one for emergency fixes. Mary said that you may find that the emergency fix is the least valuable: by the very nature of an emergency, you’ll find that process is already streamlined.

Mary closed with a very important takeaway for value stream mapping activities: The purpose of creating a value stream map is to identify the most significant source of waste in your process so that you can fix it. It is not to show how great you are. The big picture goal is to create “a reliable repeatable cycle time from need until that need is satisfied.” 

If the rest of CodeMash continues to live up to this, it will be time very well spent.

Paint Wars at CodeMash

One of our engineers, Chris Marinos, has some interesting content to discuss at CodeMash: Paint Wars.

Paint Wars is a game that Chris some of his classmates wrote for a senior project at the University of Michigan. The original game was written in C#.

There is a purpose in bringing Paint Wars to CodeMash: Chris and Mike Woelmer have been working on re-writing Paint Wars in F#. The plan was to understand how F# might provide better support for some of the computing needs for a game. Also, was F# a more productive environment? If so, for everything, or for only some tasks?

Chris will be blogging about what they learned during this process. If you want to learn more in the meantime, visit us at CodeMash.

If you want to learn why we feel this was time well spent, Anne Marsan said it best here.

The Technology Industry in 2009 (One man’s opinion)

It’s time to consider what 2009 will bring for the technology industry. It’s more than just an idle exercise: In order to stay relevant, I need to make some reasonable predictions about what I should be learning as new technology becomes available. So here’s where I’m investing my learning time, and why.

Cloud Computing

Sure, this one is easy.  The big players in our industry are spending millions to build cloud computing infrastructure. I doubt they are all wrong.

Personally, I believe it’s too early to pick winners among the major players. In fact, they may all be winners. More importantly, I want to consider what cloud computing will enable us to build.

‘Computing in the cloud’ gives us scope we’ve never had before. Scope of CPU, scope of storage, scope of geography.

CPU scope means that applications out of our reach will now be possible. Consider genetic algorithms. Larger populations will be possible, because the cost function for subsets of the population can be distributed among more CPUs in the cloud. Applications like SETI at home can easily move to the cloud. Any applications that can be scaled into multiple processes can be moved into the cloud, and can be scaled to solver larger and larger problems.

Storage Scope means that applications that need more and more storage can be considered in the cloud computing era. What can we do to solver world health problems if we can analyze data from everywhere in the world? Can we cure cancer? Can we analyze the spread of viral outbreaks with world-wide data? If we can analyze them, can we stop them? Or at least slow them?

Clearly, CPU scope and storage scope are related. Many problems that require enormous amounts of data also require tremendous CPU resources. In both cases, we’re now going to create software that can address larger and larger problems.

The elasticity of the cloud computing platforms available means that it will be easier to validate these new applications on smaller subsets of the data, or algorithms. Then, as the algorithms are proved in the small scope, more and more resources can be brought to bear on larger and larger problems.

Rich Internet Applications (RIA) grow up

Right now, the industry doesn’t have a consistent definition of an RIA application. Is it a web application? Smart client? Something new? A mix of all those things we’ve already had?

It seems that no one knows for sure. And yet, there’s clear direction from all of us that use software on the web.  The occasionally connected, or casually connected user, is important. Web applications don’t work for them. RIA applications must satisfy the user that isn’t always connected to the web. That means something different than a browser based application. It means a UI that works offline. It means a data storage model that works offline. It means real state management. It means a push model: send data to the user rather than asking the user to poll for updates as an explicit action.

Of course, not everyone will want, or be allowed to install or run a rich client. And, your users may sometimes be using borrowed machines. That means supporting a browser based experience for broader reach.

In my opinion, an RIA will come to mean an application that has several work models. The key feature will be server-based (or cloud based) data storage, multiple client based experiences, and every one of those client experiences will be as as rich as possible, based on the constraints of the delivery experience. Is that Silverlight? Flash? something new? A WPF application using WCF to communicate?  Well, yes. It means all those things.

In addition, because the data storage will be based at the server, or in the cloud, RIA applications will be, by definition, highly collaborative. You’ll be sharing data, and sharing work, with many others. The RIA applications must be capable of handling that kind of collaborative workflow.

Multi-touch goes Mainstream

Surface? Windows 7 with Multi-touch? Even the iPhone. It’s a start. The current UI metaphors are  more than a decade old. It’s hard to believe that we’ve not come up with any new ways to communicate with our machines.  I want the speech recognition from Star Trek. OK, that may be a bit too far in the future, but I do believe it’s time for something new.

Direct manipulation on the screen seems to be the most promising.

As with cloud computing, it’s more interesting to look at what these new metaphors might enable us to accomplish. Surface computing is highly collaborative. Many people can work on the same computing platform together in ways that just aren’t possible using the traditional PC (or Mac). Multi-touch is going to bring surface – like metaphors to the mainstream. In fact, it may help drive adoption enough that the price point for a surface will come down to where it becomes a mainstream computing device.

I think that as more and more applications show up for these new and different platforms, there will be more and more demand for applications that make use of these new metaphors.

Social Networking as a business tool

Most businesses are afraid of social networking applications. The current thinking is that social networks (facebook, twitter, blogging) provide a megaphone for a business’s harshest critics. That’s true, but that’s only one facet of how social networks can affect businesses. It’s also not one of the better uses of social networks for businesses.

Instead, I believe that this year, businesses will learn to leverage social networks to build customer communities, or enable customers to endorse products. In addition, businesses will learn to use social networks to keep their customers up to date on new and upcoming products, events, and more.

It’s really just another way to communicate. What seems hard for businesses is that it’s more of a two-way conversation than traditional marketing efforts. The winners will be the companies that can learn how to promote what they do in the context of a two way conversation.

Conclusions

Of course, I’m probably wrong about many of this. Predicting the future is very difficult. However, I do believe all four of these topics will be important over the next year, as businesses try to create value and compete for customers in a shrinking world economy. Software is going to command some resources, but in order to continue to grow, software will need to provide ever increasing value for ever decreasing costs. All of the topics I’ve mentioned above will help software companies create greater value.

Search

Go

Blog Group Links

Nascar style badges