|
|
These blog entries are written by industry experts and leaders. We consider this content to be a good read for any software developer or web technologist.
December 2008 - Posts
-
 It's New Years Eve and what better way to spend the night then writing up a post about the last year. The biggest change this year was the birth of my wonderful daughter Katherine, who we affectionately call Rin. It truly has been life changing and while it would be nice if she slept a little bit more than she does she is an extremely cute and happy baby who is growing and learning so quickly.
I feel good about what I was able to accomplish this year. I covered some of the progress on The Lounge in an earlier post, but basically I took over The Lounge in late december and managed to quadruple the number of publishers, venture into podcast, screencast, and RSS advertising and increase the revenue by over 5x. I launched the Ruby Room which because Ruby Row and it is doing great so far. I wish I had launched more networks this year and that will be a major focus of mine next year.
I also had the pleasure of working with Microsoft and the SDS team to write the ruby library for Microsoft SDS, it was great to not only be playing with a cool new technology like SDS but to be using Ruby with it made it even better.
Earlier this year I posted some goals so here is a quick rundown of how I did.
Goal #1: Lose Weight and get into shape.
My plan to lose weight didn't work out very well but I have high hopes for the year ahead. I never expected how much time a baby would consume and it has made it tough to get back into my running schedule. Things are starting to stabilize though and I think I will be able to get back into it and hope to run at least a 10k by the end of the year.
Goal #2: Minimize and Simplify
This isn't a great goal since it's hard to measure, but I do feel like I have minimized and trashed alot of clutter and that I have embraced simplicity in everything I attempt. (it is a constant struggle to keep things simple, but it is a very noble goal)
Goal #3: Stop Consulting
I didn't manage to do this in 2008, but my long-term contract is ending Jan 15th and I will be attempting to avoid taking another long-term contract. I am open to smaller contracts, but I don't want another 6-12 month contract and if possible I would like to avoid it completely. (more on this later)
Goal #4: Make 22books a success
Well, this definitely didn't happen. I decided I needed to focus on The Lounge and Ruby Row over doing any work on 22books. I haven't made any updates in months, but the site continues to work and has a small group of users. On the plus side the site has been running on a couple of mongrels without being restarted for 6+ months.
Goal #5: Grow The Lounge
I posted about this earlier this week, The Lounge has continued to grow and has started to surpass my expectations. I have huge plans for The Lounge in 2009.
Goal #6: Re-Launch Windows Developer Power Tools
I missed this one as well, my decision to focus on The Lounge and Ruby Row meant other ventures like this suffered.
Goal #7: Reset this blog
I didn't write a post a week, I am not even sure I wrote a post a month. I can't believe that I used to post daily back in 04, but I lived a totally different life back then... I wish I had that time now.
So while I didn't hit alot of my goals I learned some very important lessons about focus. By neglecting 22books and WDPT as well as the blog I was able to get much more done on The Lounge and Ruby Row than I would of otherwise . I am going to keep this in mind when making goals for next year.
-James
|
-
Today we launched a new ASP.NET MVC Design Gallery on the www.asp.net site. The design gallery hosts free HTML design templates that you can download and easily use with your ASP.NET MVC applications. Included with each design template is a Site.master file, a CSS stylesheet, and optionally a set of images, partials, and helper methods that support them. The gallery allows you to preview each of the designs online, as well as download a .zip version of them that you can extract and integrate into your site. The gallery allows anyone to create and submit new designs under the creative commons license. Visitors to the gallery can vote to provide feedback on them (thumbs up/thumbs down). The most popular designs show up at the top of the gallery. 
We think this will provide a useful way for developers to more easily create attractive, standards compliant, sites. It will also hopefully encourage folks to create and share designs that can be easily re-used by others. Upcoming View Improvements with the Release Candidate While on the topic of UI, I thought I'd also share a few details about some of the View-related improvements that are coming with the new ASP.NET MVC Release Candidate build that will be shipping shortly. In addition to bug fixes, the RC incorporates a number of view-specific feature additions and community suggestions. Views without Code-Behind Files Based on feedback from a lot of people, we've decided to make a change so that MVC view files by default do not have code-behind files. This change helps to reinforce the purpose of views in a MVC world (which are intended to be purely about rendering and to not contain any non-rendering related code), and for most people eliminates unused files in the project: 
With the ASP.NET MVC Beta, developers could eliminate the code-behind file by using the CLR syntax for generic types in a view's inherits attribute, but that CLR syntax is (to put it mildly) pretty undiscoverable and hard to use. The ASP.NET MVC team was able to combine a few extensibility features already in ASP.NET to now enable the standard VB/C# language syntax within the inherits attribute with the ASP.NET RC build: 
One other nice benefit of not using a code-behind file is that you'll now get immediate intellisense when you first add them to the project. With the beta you had to do a build/compile immediately after creating a view in order to get code intellisense within it. The RC makes the workflow of adding and immediately editing a view compile-free and much more seamless. Top-Level Model Property on Views With previous builds of ASP.NET MVC, you accessed the strongly typed model object passed to the view using the ViewData.Model property: 
The above syntax still works, although now there is also a top-level "Model" property on ViewPage that you can use: 
This property does the same thing as the previous code sample - its main benefit is that it allows you to write the code a little more concisely. HTML/AJAX Helpers Now Enable Expression Syntax One of the requests a few people have asked for is the ability to use strongly-typed expression syntax (instead of using strings) when referring to the Model when using a View's HTML and AJAX helper objects. With the beta build of ASP.NET MVC this wasn't possible, since the HtmlHelper and AjaxHelper helper classes didn't expose the model type in their signature, and so people had to build helper methods directly off of the ViewPage<TModel> base class in order to achieve this. The ASP.NET MVC RC build introduces new HtmlHelper<TModel> and AjaxHelper<TModel> types that are exposed on the ViewPage<TModel> base class. These types now allow anyone to build strongly-typed HTML and AJAX helper extensions that use expression syntax to refer to the View's model. For example, I could build a (very simple) strongly-typed "TextBox" helper method using the code below: 
And then use it within any of my views to bind against a Product model object like so: 
Visual Studio will provide full intellisense for the strongly-typed expression syntax when working against the View's model in the source editor in this way: 
Note: the HTML helper extensions in the core ASP.NET MVC V1 assembly will still use the existing (non-expression based) syntax. We are then planning to add expression-based versions to the MVCFutures assembly. You can of course also add your own helper methods (using either strings or strongly-typed BLOCKED EXPRESSION. All of the built-in helper methods can also optionally be removed (because they are extension methods) if you want to replace or override them with your own. Scaffolding Support The ASP.NET MVC RC build includes automatic "UI scaffolding" support when creating views using the new ASP.NET MVC "Add View" command inside Visual Studio. The scaffolding support enables the automatic generation of views against any .NET type or object - meaning it can work against POCO classes, LINQ to SQL, LINQ to Entities, NHibernate, SubSonic, LLBLGen Pro, or any other object model. The scaffolding engine uses reflection to retrieve the public shape of a View's model type, and then passes it to a scaffolding template to populate appropriate markup based on it within the view being created. For example, assume we have a ProductsController class and want to create an "Edit" action on it to display an edit view of a particular Product. Using the RC build we can right-click within our "Edit" action method and choose the "Add View" context menu command like so: 
Within the "Add View" dialog we can then indicate that we are passing a "Product" type to our View: 
We can indicate that we want an "Empty" view template created (like above), or indicate that we want VS to automatically scaffold a form "Edit" view for the Product object we are supplying: 
If we choose the "Edit" template VS will automatically generate a file for us that has the appropriate HTML and validation helpers to create an editable form view: 
We can then run the application and immediately get edit UI: 
We can then go in and change the generated edit.aspx file however we want to tweak/customize it. One of the really nice things about the scaffolding system we are shipping is that it is implemented using Visual Studio's built-in T4 code generation system (Scott Hanselman has a nice post about this here). The "List", "Edit", "Create" and "Details" templates we ship with ASP.NET MVC can all be completely customized or replaced with T4 templates of your own (or downloaded from the ASP.NET MVC Design Gallery). So if you have your own particular way of creating HTML, or want to use custom HTML helpers (for example: strongly-typed expression based ones) you can update the default templates and the scaffolding system will use them going forward. We are planning to enable the templates to be overriden both on a machine-wide level, as well as on a per-project level (so that you can check-in application-specific scaffolding templates under source control and share them across a team). MSBuild Task for Compiling Views By default when you do a build on an ASP.NET MVC project it compiles all code within the project, except for the code within view files. With the ASP.NET MVC Beta you had to roll your own MSBuild task if you wanted to compile the views. The ASP.NET MVC RC build now includes a built-in MSBuild task that you can use to include views as part of the project compilation process. This will verify the syntax and code included inline within all views and master pages for the application, and give you build errors if it encounters any problems. For performance reasons we don't recommend running this for quick compiles during development, but it is convenient to add to particular build configuration profiles (for example: staging and deployment) and/or for use with Build or CI (continuous integration) servers. Other ASP.NET MVC Release Candidate features coming Above is a short list of some of the view-specific functionality coming with the release candidate build. There are many other features and requests coming with the RC as well including: IDataErrorInfo support to enable models to surface validation error messages, as well as richer error validation extensibility to enable you to use your own approach to surface model validation errors to ModelBinders (the IDataErrorInfo support is built on top of this); new FileResult and JavaScriptResult ActionResult types (allowing you to more easily download files as well as executable JavaScript to browsers); built-in jQuery -vsdoc intellisense support; refactored AccontController support to enable easier unit testing and extensibility with form login scenarios; a variety of project template improvements, more extensibility everywhere; lots of bug fixes; and a few other cool features I'll blog about later once the RC is out. We'll be releasing the ASP.NET MVC Release Candidate in January. Our plan is to have that build be ASP.NET MVC V1 API and feature-complete and have zero known bugs. We'll give people a short period to upgrade to it, give it a good tire-kicking, and report any last minute issues they find. We'll then ship the official V1 release shortly after that (so not far off now). Hope this helps, Scott 
|
-
Aries (March 21-April 19): Avoid committing yourself to the next project at work. It’s going to become a death march, and you know it. Save your skills and energy for some open source hacking. Taurus (April 20-May 20): You’ve been flirting with functional programming and now it’s time to take the plunge. Free your soul of side-effects and embrace a monad. It will make you feel good. Gemini (May 21-June 21): You are entering a period of introspection. For pair programming, it’s best to hook up with Cancer. Avoid Pisces, because you know you’ll bicker over inheritance versus composition until someone gets hurt by a fast-moving keyboard. Cancer (June 22-July 22): Your creative juices are flowing. Color coordinated themes will jump from your mind to fill the soft, supple curves of the rounded rectangles your customers visually crave. Who said developers can’t design a UI? Not you! Leo (July 23-Aug. 22): You might start feeling detached from the rest of the team. Now is the time to randomly refactor code that someone else wrote in the name of collective code ownership. You might spark a new relationship! Virgo (Aug. 23-Sept. 22): You might need to apply some fundamental design patterns to find the elegant solution you seek. Repository. Abstract Factory. Visitor. Pick 2.
Libra (Sept. 23-Oct. 23): There are lots of meetings in your future. Some of those future meetings will be meeting to discuss future meetings (the meta-meeting meeting). Good for you the gaming market for cell phones is hitting its stride. Scorpio (Oct. 24-Nov. 21): Don’t get frustrated with what your future holds. Broken builds, failing tests, and bug reports like “I think it broke when I click Submit, or something”. Be positive and live stress free. Consider taking up Yoga. Sagittarius (Nov. 22-Dec. 21): You should start taking security seriously. SQL injections, code injections, script injections, header injections. Everyone is out to get you, and it’s only a matter of time. Only the paranoid survive. If you can trust a Scorpio, you might find a mentor. Capricorn (Dec. 22-Jan. 19): Time for a change. If you’ve been using a statically typed language, then try something dynamic. If you’ve been using a dynamic language, then try something static. If you do both, then try Malbolge.
Aquarius (Jan. 20-Feb. 18): To you, software is a craft. You always have your eye on an impossible star, but you reach anyway. The vague predictions of horoscopes drive you crazy, but no one ever accused them of being a science, eh? Pisces will be your friend.
Pisces (Feb. 19-March 20): Monkeys with keyboards – that’s what you may be thinking about your team when the next project starts. When life gives you monkeys – make bananas. Or something like that. Watch out for Sagittarius. 
|
-
One of the first things I do when creating a new ASP.NET web application is create a custom base page class and add a handful of useful methods I've used in other projects. My latest article on DotNetSlackers.com shares four helpful features that you can add to your base page class:
- Display a JavaScript Alert
- Recursively Search the Control Hierarchy
- Record Page Execution Times
- Set the Page's Title
Read more at: Four Helpful Features to Add to Your Base Page Class.
|
-
On December 16th 2007, a year ago today, I took over The Lounge Advertising Network from my buddy Kevin F. Here is a quick summary of the first year:
- Grew from 10 publishers to 40
- Launched the Small Publishers Room
- Launched the first .NET focused RSS advertising offering with the RSS Room
- Launched the Podcast and Screencast rooms
- Started the year with 2 advertisers, sold out all rooms for the first time in July.
I also have a couple of new things to announce today. The first is a couple of new publishers who actually joined last month but I never blogged about. Both have joined the Small Publishers Room and I think make great additions.
![]()
Kevin Pang writes a great blog and you will quickly notice he loves creating lists.
![]()
Ken Egozi is very active in open source. He is a committer to the Castle project and helps out on the NHibernate mailing list and wiki.
![]()
The second announcement is a new room called the Open Source .NET Room. This room will be a way for open source projects and developers to make money from advertising without resorting to large banner ads or google adsense. I was hesitant to start this room because I don't want to be seen as profiting off of open source, so all the proceeds of this room are going to fund the servers that run ideavine.net. If you have an open source project and you would like to be a part of this room please email me (javery at this domain).
I also have another new room that will be launching in the new year, but I am not yet ready to make that public.
Its been an amazing year and I hope next year is even better.
-James
|
-
The Unit of Work (let’s call it UoW) is another common design pattern found in persistence frameworks, but it’s new to many .NET developers who are just starting to use LINQ to SQL or the Entity Framework as the “new ADO.NET”. First, the obligatory P of EAA definition: A Unit of Work keeps track of everything you do during a business transaction that can affect the database. When you're done, it figures out everything that needs to be done to alter the database as a result of your work. Please think of a “business conversation” if the phrase “business transaction” makes you think of database transactions with BEGIN and COMMIT keywords. Although a UoW may use a real database transaction, the idea is to think at a slightly higher level. When your software needs to fulfill a specific business need, like process an expense report, the unit of work will record all the changes you make to the objects involved in processing the expense report and save those changes to the database. The unit of work is like a black box recorder. DataSets and the UoW If you’ve been using ADO.NET, then it turns out you’ve already been using a UoW. The ADO.NET DataSet offers a good UoW implementation. You can take some records from the database and shove them into a DataSet, then make changes to the data during the course of a business conversation. When the conversation is complete you can use a table adapter thingy to save the entire batch of changes into the database. The DataSet and adapter combination gives you concurrency checks and the ability to undo changes. You can even take a DataSet and send it on an epic journey outside of your app domain, outside your process, or even outside your machine, and wait for the DataSet to return at some future point in time. During its journey, the DataSet will faithfully continue to record changes the adapter can save. DataSets have drawbacks, however, and I’m not going to resurrect that tired conversation. Suffice to say there is a reason that many developers have, for many years, been using persistence frameworks that offer a higher level of abstraction. Persistence Frameworks and the UoW The UoW implementation varies widely among the different persistence frameworks. LLBLGen has a serializable class explicitly named UnitOfWork that supports field level versioning, while NHibernate hides a UoW implementation behind an ISession interface. In LINQ to SQL the UoW centers around the DataContext, while in the Entity Framework the UoW centers around the ObjectContext (let’s just refer to both as the Context objects). I’m not going to compare and contrast the different implementations – I’m just pointing out that most frameworks provide a UoW, but the features can vary. It is the L2S and EF implementations that the rest of this post will discuss because they often trip up the ADO.NET developers who make assumptions about the how the Context will behave. With ADO.NET DataSets, every query sent to the database will retrieve the latest data, but this isn’t always true with the Context objects. [TestMethod]
public void Context_Is_Isolated_From_Outside_Changes()
{
using (var ctx = new MovieReviewsContext())
{
var movie = ctx.Movies.Where(m => m.ID == 100)
.First();
// make a change
movie.ReleaseDate = new DateTime(2007, 1, 1);
SimulateOtherUsersWork();
// decide we want to refresh from database
movie = ctx.Movies.Where(m => m.ID == 100)
.First();
// but we don’t see the new data!!!!!!!!!
Assert.AreEqual(2007, movie.ReleaseDate.Year);
}
}
public void SimulateOtherUsersWork()
{
using (var ctx = new MovieReviewsContext())
{
var movie = ctx.Movies.Where(m => m.ID == 100)
.First();
movie.ReleaseDate = new DateTime(2008, 1, 1);
ctx.SaveChanges();
}
}
Remember that the context objects implement an identity map. When the second LINQ query goes to the database and fetches the Movie record with an ID of 100, the context will consult its identity map and see it has already created an object to represent the Movie with an ID of 100. The context simply returns a reference to the existing object instead of creating a new Movie entity – meaning the context effectively ignores any changes that have been made in the database.
Some developers will think this is a bug (you mean I can’t get fresh data?), but the context objects are designed to live for a single unit of work - a single business conversation. During this time they also isolate your entities from changes that were persisted in different business conversations. Imagine passing an ExpenseReport for 1000 paperclips through business validation logic and submitting changes – only to have the ExpenseReport contents morph into 1000 voodoo dolls immediately after validation because of some query that was executed implicitly. Practically speaking, many applications will never run into the problem of picking up unintended changes, but this isolated behavior is on by default and the consequences catch more than a few people off guard.
Why Is This Important?
Using LINQ to SQL’s DataContext or the Entity Framework’s ObjectContext for more than one business conversation is usually the wrong thing to do. These classes are designed to support a single unit of work because of the way they track changes and use an identity map. If you keep them around too long and do too much work, then the identity map will grow too big, and the data will become too stale.
For desktop applications it is common to start a unit of work when a dialog opens, and complete the unit of work when the dialog closes (context per view). On the server side it is common to start a unit of work when an HTTP request begins processing, and close the unit of work when the HTTP request ends (context per request).
Of course, not every business conversation can complete in a single request or a single dialog box. Managing “long winded” conversations with EF or L2S is tricky, unfortunately, and it’s part of the debate you’ll see in blogs these day. We’ll have to address this issue in a different post. 
|
-
WPF, AJAX, or Silverlight RIA?
I gots more options than a block of Velveeta.
XBAP? XCOPY? XAP files are just ZIPs!
Let me run my bits on your silicon chips. When I started to think of what my XM Radio Player would look like, I could only picture two things: - Rectangles with rounded corners.
- Gradient color fills.
Given this detailed initial vision I knew I had to pick WPF. What about Silverlight? I considered Silverlight, but three factors kept me away. One issue would be the cross domain calls to the XM Radio domain. Although XM does put a crossdomain.xml file on their server, it only allows calls from *.xmradio.com, which meant I’d have to deploy a web service just to thunk calls over to XM, which isn’t hard, but added one additional piece of complexity. Secondly, I’ve been up to my ears in Silverlight a few times this year and wanted to try something new. Thirdly, it’s been some time since I’ve written a desktop app and I wanted to have some fun, and also take a look at Prism. 
|
-
There are a couple of important patterns in play when you use a persistence framework. These patterns have been around for quite some time but are relatively new to .NET developers who are jumping into LINQ to SQL and the ADO.NET Entity Framework. The first of these patterns is the Identity Map pattern. There are various aliases for this pattern. Some frameworks call it the “identity cache” while others have an “entity uniqueing” implementation. Using examples as tests, we can describe its behavior with a passing test (the code is written with the Entity Framework, but you could substitute a LINQ to SQL DataContext (or an LLBLGen Context, or an NHibernate Session) and observe the same behavior – the point being that identity maps can be found in almost every framework for persisting objects). [TestMethod]
public void Context_Implements_An_Identity_Map()
{
var context = new MovieReviewsContext();
var movie1 = context.Movies.Where(m => m.ID == 100).First();
var movie2 = context.Movies.Where(m => m.ID == 100).First();
var movie3 = context.Movies.Where(m => m.ID > 98 && m.ID < 105)
.ToList() // bring them all into memory
.ElementAt(1);
Assert.IsTrue(Object.ReferenceEquals(movie1, movie2));
Assert.IsTrue(Object.ReferenceEquals(movie1, movie3));
}
The test will execute three queries against the database. All three queries will retrieve the movie record with an ID of 100, with the last query also pulling some additional records. No matter how many times we query the database, even using different queries, exactly ONE object represents the movie with an ID of 100. The context uses an identity map to track the entities it creates. The mapping information we provide to the framework allows the context to know which property uniquely identifies each entity (the integer ID property in the case of a a Movie).
You can think of the identity map as a Dictionary<int, Movie>. Each time the context sees a resultset from the database, it checks the dictionary to see if it already created an entity for the data. If so, it just returns a reference to the entity it already created. This is different than the behavior of ADO.NET DataSets, where you can issues three queries for Movie.ID == 100 and get back three distinct DataRows. Also, its important to know that each DataContext/ObjectContext in LINQ to SQL/Entity Framework maintains its own identity map. [TestMethod]
public void Each_Context_Has_Distinct_Identity_Map()
{
var context1 = new MovieReviewsContext();
var context2 = new MovieReviewsContext();
var movie1 = context1.Movies.Where(m => m.ID == 100).First();
var movie2 = context2.Movies.Where(m => m.ID == 100).First();
Assert.IsFalse(Object.ReferenceEquals(movie1, movie2));
}
Why Is This Important?
Some people I’ve talked to don’t like the behavior of the identity map, or are surprised by the behavior. But, imagine if we could create two objects from the same context and both objects represent the movie with an ID of 100 – and then we start making changes to those objects. Which object should “win” when we save changes to the database? This is the scenario Fowler analogizes in his definition of the identity map pattern.
An old proverb says that a man with two watches never knows what time it is. If two watches are confusing, you can get in an even bigger mess with loading objects from a database. If you aren't careful you can load the data from the same database record into two different objects. Then, when you update them both you'll have an interesting time writing the changes out to the database correctly.
Identity maps enforce object identity in memory in much the same way that a database enforces row identity in a database table using a primary key value.
Not every application needs the safety net afforded by an identity map implementation, but, when one is in place you need to be aware of its ramifications. We’ll cover these in a future post on another important pattern – the Unit of Work. 
|
-
Just to make sure everything was as easy as it looked in Fiddler – I wrote a quick and dirty piece of throwaway code to see if I could programmatically login to XM and play a stream of music with Windows Media Player. It was ugly, but … public void Can_Start_Media_Payer_With_Xm_Stream()
{
var cookies = new CookieContainer();
// step 1: get auth cookie
HttpWebRequest request = WebRequest.Create("http://xmro.xmradio.com"
+ "/xstream/login_servlet.jsp")
as HttpWebRequest;
request.CookieContainer = cookies;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
var requestStream = request.GetRequestStream();
var loginData = Encoding.Default.GetBytes("user_id=*******&pword=******");
requestStream.Write(loginData, 0, loginData.Length);
requestStream.Close();
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
var data = ReadResponse(response);
// ...
// step 4: get player URL for channel 26
request = WebRequest.Create("http://player.xmradio.com" +
"/player/2ft/playMedia.jsp?ch=26&speed=high")
as HttpWebRequest;
request.Method = "GET";
request.CookieContainer = cookies;
response = request.GetResponse() as HttpWebResponse;
data = ReadResponse(response);
Regex regex = new Regex(
"<param\\s*name=\"FileName\"\\s*value=\"(.*)\"",
RegexOptions.IgnoreCase);
string url = regex.Match(data).Groups[1].Value;
Process.Start("wmplayer.exe", url.ToString());
}
… it worked!
The most difficult piece was digging out the FileName parameter from the playMedia servlet response. The HTML response was almost, but not quite, XHTML complaint. LINQ to XML wasn’t an option, unfortunately, so I used a regular expression. I don’t write regular expressions often enough to build them without a lot of help, which is why I keep a copy of Roy Osherove’s Regulator around. I could paste a copy of the servlet response into Regulator and then iteratively hack at a regular expression syntax until something worked and I cold stop cursing. In a couple weeks I’ll have no idea what stuff like \”(.*)\”. means anymore, which is both good and bad at the same time.
It’s always liked the idea of looking at the high risk areas of a project and writing some throwaway code to see if a solution is technically feasible. Some people call this a “spike” and as I’ve said before – you never want to let spike code into production (but you never want to really throw the code away, either).
Screen Scraping
The regex in the above code is “scraping” data from the HTML. Screen scraping is nearly as old as computers themselves – in fact it’s one of the earliest forms of interoperability. One way to get data from a mainframe is to capture the data from the mainframe’s display output on a terminal– thus the term “screen scraping”. Not that I’ve ever written a terminal scraper, but I’ve seen them in action and I wouldn’t be surprised if 90% of all Fortune 500 companies that have been in business for more than 30 years are still using screen scraping somewhere for enterprise application integration.
Something I have done quite a bit is screen scraping for the web. “Web scraping” (as some call it) has a slight negative connotation these days as it can be used for nefarious purposes – like the spammers who harvest email addresses from web pages. There have also been some legal battles when people screen scrape to repurpose someone else’s content. But generally speaking - screen scraping isn’t inherently bad. In this case I’m only using scraped data to build a custom player and listen to music with my paid subscription.
With the .NET libraries there are really just three key areas to focus on:
- Properly formulating the request
- Managing cookies
- Parsing the response
Properly formulating the request was relatively easy in this scenario, even with the credentials in the POST data. Comparing what your program is sending to the server with what the web browser sends to the server using a tool like HTTP Fiddler is the best way to troubleshoot problems. Some scenarios aren’t as easy. For instance, if you need to programmatically post to an ASP.NET web form you need the proper hidden field values for stuff like viewstate and event validation, which means you need to parse these hidden fields from a previous GET request for the same web form.
If a web site requires cookies to work properly (as XM does), then you’ll find that managing cookies with the .NET libraries is tricky – but only because it doesn’t happen by default. However, all you need to do is instantiate an instance of the CookieContainer class and attach this same instance to all of your outgoing web requests. .NET will then take care of storing cookies it finds embedded in a server’s response, and transmitting those cookies back to the server on subsequent requests. The framework knows all about cookie domains and secure cookies, so you don’t need to manually finagle HTTP headers at all – just use the container.
Parsing the response is generally where you invest most of your time and feel the most pain. There are managed libraries like the Html Agility Pack that can help, and 1000 other solutions from String.IndexOf, to regular expressions, to Xpath queries over XHTML complaint markup. The basic problem is that sites generally don’t design their web pages for machine-to-machine interaction and they can change their markup at any time and break your scraping logic. No matter how robust your scraping logic is – nothing short of a perfect AI can keep your logic from breaking. It’s always better to stick with official interoperability endpoints that use SOAP, REST, or some other form of structured response that is intended for machines. It’s strange that we are almost in the year 2009 and we will still be using strings and regular expressions to interop with most of the web for the foreseeable future.
Next Steps
Knowing that a custom player was possible, the next step was to pick a platform to build on.
Stay tuned… 
|
-
Over Thanksgiving weekend Fredrik Normen asked an interesting hypothetical question: What would you be doing today if computers didn't exist?
I've been programming since age 10 when my family procured its first personal computer and my dad showed me how to use GW-BASIC and your imagination to make computer games. So it's hard to imagine a life without computers. (Heck, it hard to imagine life without the Internet, yet close to half of my computing lifetime was spend disconnected. I guess I have a poor imagination.)
If computers didn't exist I would probably be employed in one of three fields:
- The sciences - in high school I had an interest in biology and physics and, for a brief time, actually considered majoring in Biology at University before deciding that computer science would be a more interesting and enjoyable path. If computers had never existed I could see myself going down this path, perhaps being an 'in the field' biologist or, if the math didn't kill me, a physicist.
- Education - I've always enjoyed teaching and tutored younger kids in science and math during high school to earn some spending money. I doubt I'd have the patience or fortitude to be a full-time public teacher, but teaching in a higher education facility or personal tutoring could be something I'd do for a living.
- Writing - I started writing stories (longhand) when I was in elementary school, and didn't stop writing fiction until college. If computers didn't exist perhaps I'd be a technical writer, writing the instruction manual for abacuses and slide rules.
What would you be doing today if computers didn't exist?
|
-
 The Jenkins video always makes me laugh. It’s the short and sad tale of a man who doesn’t have the patience for planning, nor the stamina for statistical analysis. He’s a man who leaps into action without thought of consequence. To Leeroy, there are no 50,000 foot views, no discussions, and … no tomorrow. In the video you can watch as Leeroy’s guild is decimated by dragons who take advantage of the chaos provided by Leeroy’s thoughtless actions.
You might be thinking I’m going to liken Leeroy to the developer who writes 200 lines of untested code inside a method named “SubmitOrder” before his team has even left the conference room where they are discussing what the software is supposed to do – but I’m not going to make that analogy. Well, not that exact analogy. I think there is a time and a place to get some Leeroy into your life. I’m always amazed at email threads where participants debate the possible outcomes of psuedocode when it would easier to just jump in and let the computer decide the outcome. When in doubt – just write the code (preferably with a Leeroyesque shout). Thought experiments are for physicists! 
|
-
I'm flying out later today on a pretty intense business trip (22,000 miles, 5 countries, 3 continents, 1 week, no sleep... :-), so my blog activity over the next week and a half will be pretty light. To keep you busy till I return, here is the latest in my link-listing series. Also check out my ASP.NET Tips, Tricks and Tutorials page and Silverlight Tutorials page for links to popular articles I've done myself in the past. ASP.NET ASP.NET Dynamic Data ASP.NET AJAX ASP.NET MVC Visual Studio WPF / Silverlight Hope this helps, Scott 
|
-
I’ve had an XM radio subscription for a couple years now but I’ve never been entirely happy with any of the options for XM Radio Online. XM’s online player has changed very little over the years, and it is missing many simple features you can find in other third party players and real radio hardware – like artist and song notifications. A couple days ago I started to think about just writing my own XM radio player for the heck of it. The first step was to see what goes on between the browser and the server when streaming music. Note: this is an attempt to build a legitimate custom player to use with an XM subscription – not an attempt to get XM programming for free. Fiddler Fiddler is my tool of choice for web traffic sniffing. If you are just looking for tools to monitor the JSON payload sizes for an AJAX-ified web page, then there are a million tools and plug-ins with this functionality. Fiddler goes well beyond the entry level sniffing tool and touts itself as an HTTP debugger. You can set breakpoints, tamper with requests, build requests (from scratch, or by dragging and dropping a request from the list of previous requests), force requests to trickle (to simulate slow networks – great for testing SIlverlight splash screens), and extend Fiddler with managed code. There is also a CustomRules.js file that one can use to quickly extend and customize Fiddler using JScript.NET. As an example, the following code will place a new “Hide CSS” option in the Fiddler Rules menu and declare a variable to store the state of the menu item: public static RulesOption("Hide CSS requests")
var m_HideCSS: boolean = false;
To hide CSS requests when the menu item is checked, add the following to the OnBeforeRequest function: if(m_HideCSS) {
if(oSession.uriContains(".css")) {
oSession["ui-hide"] = "hiding css";
}
}
There are dozens of other useful rules in the Fiddlerscript cookbook.
 Perhaps the most unknown feature in Fiddler is the QuickExec box (or at least I didn’t know about QuickExec until recently). QuickExec is the little black box at the bottom of the recorded session list. The box serves as a command line interface to Fiddler, and you can use QuickExec to filter the session, perform string matching, set breakpoints, launch a web browser, and more. There is a video demonstration for QuickExec available, and a list of default commands. If you use Fiddler regularly it’s worth your time to learn QuickExec and find the shortcuts that make your job easier.
Fiddler and XM
It only takes a few minutes with Fiddler to figure out the four important endpoints for XM radio online.
- A “login servlet” that requires a POST with user credentials in the payload. A successful login will respond with an auth cookie that needs to be sent along on subsequent requests. In ASP.NET we’d call this forms authentication.
- A “channel data” servlet that returns the available XM channels in JSON.
- A “now playing” servlet that returns the artist and song currently playing on each XM channel (again in JSON).
- A “play media” servlet that accepts a channel number in the query string and returns HTML. Buried in the HTML is an object tag that instantiates media player and includes the URL to the music as a parameter (actually the URL parameter points to an endpoint that returns an ASX playlist which media player understands – the actual mms:// endpoint is in the ASX). The URL includes what looks like an encrypted auth token - the auth cookie isn’t required anymore, but the streaming server can still determine if the user is authorized to steam music.
This was enough information to decide that building a custom player isn’t terribly difficult. The next step was to write some .NET code just to prove there wasn’t any technical hurdles in walking through these 4 endpoints. We’ll look at that in Part II. 
|
|
|
|