BusinessRx Community

Between work and diaper changes I've been reading Michael Coles's book Pro T-SQL 2008 Programmer's Guide, and found this little gem (pg. 527-528):

The OUTPUT Clause
You can use the OUTPUT clause with the the INSERT, UPDATE, DELETE and MERGE DML statements. ... The OUTPUT clause returns information about the rows affected by the DML statements that can be useful in comparing preupdate and postupdate data, or for troubleshooting and logging purposes. ... You can use the OUTPUT clause to output a SQL result set like that returned by a SELECT statement, or you can combine OUTPUT with the INTO clause to output rows to a table or a table variable.

This feature is supported in T-SQL 2008, but was initially added to Microsoft SQL Server 2005. And here it is three years later and I'm just learning about it!

One use of the OUTPUT clause is to grab the just-inserted IDENTITY column value:

INSERT INTO TableName(ColumnList)
OUTPUT inserted.IdentityColumnName
VALUES(Values)

The above will return the just-inserted IDENTITY value as a result set, just as if you had followed an OUTPUT-less INSERT statement with the statement:

SELECT SCOPE_IDENTITY()

You could use the OUTPUT statement to return information about the rows affected by an UPDATE. For example, in these tough economic times you might need to increase prices by 20% for all products that cost less than $10.00. The following statement performs the described update and returns those products whose prices were increased, showing both their old price and their new price:

UPDATE Products SET
    Price = Price * 1.20
OUTPUT inserted.ProductID, deleted.Price AS OldPrice, inserted.Price AS NewPrice
WHERE Price < 10.00

This UPDATE statement will modify the data and return a three-column result set with a row for each modified product along with its preupdate price (deleted.Price) and its post-update price (inserted.Price).

Pretty neat, eh?

In fact, You can string these DML statements together, so you do an UPDATE with an OUTPUT whose results are then automatically INSERTed into another table (such as an audit table). The OUTPUT statement feels like in-place triggers (kind of like how Common Table Expressions are akin to in-place views).

I'll have to write an article on the OUTPUT statement on 4Guys one of these days...

Further Reading....

• The OUTPUT Clause Technical Documentation
• Retrieving the Just-Inserted ID of an IDENTITY Column Using a SqlDataSource Control
• Common Table Expressions (CTE) in SQL Server 2005

A decision all independent software developers must make early on in their career is when to bill a client and under what terms. This decision, of course, is not the developer's alone to make - it must be discussed and agreed upon by the client, as well. Similarly, a developer must decide how to charge: by hour or by project. Over the years I have received numerous e-mails from fellow developers who are contemplating striking it out on their own, and wonder what suggestions or advice I have for billing.

My general policy is as follows:

Software Development:
For software development I charge per hour. I have never taken on a client and billed per project, and likely never will. My primary concern with billing per project is that my estimation or the client's estimation may be off base, which hurts one of us. Also, when billing by the hour there is no concern about scope creep or other things tacked on by the client late in the game in an attempt to get more bang for his buck. Billing per hour is ideal for the software developer, but customers often prefer a fixed price for the project because it puts a cap on the amount it will cost. A new customer does not know what kind of work they can expect from a developer per unit time. To ameliorate this concern, I usually propose a very simple, short feature or milestone for new clients, so that they can get a good gauge of how long it takes me to complete a task, and what level of quality they can expect. I have also given price caps for customers on tight budgets, but this is more the exception than the rule.

Another reason that I propose a very short and simple first project for new clients is because I require payment in full and up-front for new customers. This sours some customers and I've certainly lost some business because of this policy. The way I see it, this requirement serves as a vetting process. It shakes out customers who may have shaky cash flow or who are going to later nickel and dime me for the work performed or, worst yet, not pay at all. Once I've had some time to work with a client and have established a good working relationship, I will relax these conditions and bill on a monthly basis (or more often, if requested). If I have not yet established a relationship and do not fully trust the client, I continue requiring payment up front and in full. When I finally end up billing on a monthly basis, my terms are Net 15 (that is, I require payment within 15 days of invoice), but I have some customers who much prefer Net 30 and once that relationship and trust is established I'm willing to agree to those terms.

I find this process of establishing trust by requiring payment up front at first, and then billing monthly with Net 15 terms to be a surefire approach to ending up with solid clients. To ensure prompt payment, some independent software developers use tactics like hosting the application on their server and not sending the code to the client until payment is received in full. Another, more questionable approach, is to put a backdoor into the application so that if the client does not pay the software developer can "break into" the application and "turn it off" until payment is received. I dislike both of these techniques as it frames the working relationship in adversarial terms.

I have waived or relaxed these requirements - payment up front for new clients, Net 15 terms, etc. - for very large or very established customers, such as regional businesses with thousands of employees, or Fortune 500 companies. These types of clients are more the exception than the rule, though, as the vast majority of my clientele are small businesses. And some companies or industries have Net 60 terms and won't make any exceptions.

Training
Like with software development, I require payment up front or on delivery of services for new clients. I will make exceptions for very large and established clients, but only sparingly. My hesitancy here in due in part to a negative experience with a local training company that was very late with payment. I ended up taking the client to small claims court, but was paid in between the time they were served and when the court date was set. (My decision to go to small claims court and the resulting process is summarized in My Small Claims Court Experience.)

While I always charge by the hour for software development, I charged for training services using a variety of models. For personal training I usually charge hourly, as it makes the most sense. In this way the customer has direct control over the costs and knows the precise charge for another hour or day's worth of training. For classroom events I usually charge a per-day rate, which depends on a number of factors, including:

• Class size
• Number of days of instruction
• The location relative to my home - is the classroom setting five miles away, or across the country?
• The material used and the topics covered - am I given the course-wear to teach or do I need to create it?

When a potential customer asks for a training quote, I use the above criteria to judge how long it will take me to create the course-wear (if needed) and to prepare for the class, how much travel time there will be (if any) and the total hours of instruction. I then multiple that number by my usual hourly rate for software development, then multiply that by a number between 1.0 and 2.0. This coefficient depends on the size of the class (the more students, the higher the number) and based on how excited I am to teach the class. If I'm teaching a two-day class to 5 smart devs who work in a fun company that's headquartered on the beach, and we're having a luau at the end of the second day, then that number will be a lot closer to 1.0 than if it's a class of 25 devs at a stuffy corporate office and I have to wear a suit each day. The figure derived from that equation, plus any travel and entertainment costs, are what I quote the client.

Writing
Unlike training and software development, it's hard to get a client to pay you by the hour for an article, tutorial, or book. Also, in my experience, it's rare to be paid by the word. Usually publishers have a pre-determined rate they pay for a written piece with a minimum word count. For print media - books and magazines - there is also a maximum word count. The pre-determined rate for magazines and websites is usually a dollar amount; for books it's a royalty rate and advance. (For a breakdown of compensation on writing a book, see The Economics of Writing a Computer Trade Book.)

Also unlike training and software development, it's virtually impossible to get a client to pay you anything up front. Book publishers offer advances, sure, but they don't start rolling in until you have completed a significant portion of the work. (This is true for computer trade authors; I'm sure Stephen King gets his advance whenever he wants.) As a result, as a write you have to be willing to get paid after the article or tutorial or book is written and published. As a result, I take a bit of a gamble if I write for a small publisher, small magazine, or small website. Having been bitten in the past, these days I rarely write for small operations unless I know the people running them and trust that I'll be compensated. But if a small publisher contacts me out of the blue and wants me to write five chapters for a book, I'll pass.

In Summary...
What's infinitely more important than the decision you make as when to you bill your clients and what terms to use, is to have very frank and clear communication before any work begins. It is imperative that you discuss with your client when you will bill and when you expect payment. They may agree, or they may want to change the terms. But what's vital is that there is a clear understanding. I've talked with many independent software developers who were too uncomfortable bringing the subject up or just assumed that the client understood and agreed with implicit payment terms, and as a result there was no communication prior to the work. This can lead to an unhappy client if his expectations of the payment terms were different than yours.

Also, if you have sent an invoice to a client and he has not paid by the time the invoice is due, do not hesitate to contact the client and politely inquire about payment. And, lastly, if a client becomes severely overdue, do not hesitate to stop working for him. You are not obligated to work for free.

Someone asked me why LINQ operators return an IEnumerable<T> instead of something more useful, like a List<T>. In other words, in the following code:

List<Book> books = new List<Book>();
// ...
IEnumerable<Book> filteredBooks = 
    books.Where(book => book.Title.StartsWith("R"));

... we started with a List<Book>, so why isn’t the Where operator smart enough to return a new List<Book>, or modify the existing list by removing books that don’t match the Where condition?

Let’s talk about modifying the original list.

I hope you’ll agree that it would be odd for a query to modify a data source. Imagine sending a SELECT statement to a database and finding out later your SELECT removed all but one record from a table. Although the Where operator is just a method call that could change the underlying list of books, it’s better to return something new and leave the original list intact. You won’t find any LINQ operators that modify input , and this behavior produces many benefits. One obvious benefit is that you can think of the above code as a query, and it won’t surprise you by removing books from your original list.

What about creating a new List<T>?

It turns out that creating a new list can be quite expensive, but only because we often don’t need a List<T> returned. Think about the number of lists created in the following query (if each operator created a new list).

var filteredBooks =
        books.Where(book => book.Title.StartsWith("R"))
             .OrderBy(book => book.Published.Year)
             .Select(book => book.Title);

Here we would have three lists created (one each by the Where, OrderBy, and Select operators). We’d only needed a single list of titles as the result, but since the operators cannot modify their underlying data source (for the reasons outlined above), they would be forced to each create a new list, and most of that work would be wasted as two of the lists are immediately discarded.

Imagine if you only needed to count the number of books whose title starts with the letter R – in that case you wouldn’t want any of these lists being created and destroyed when you computed the result. It would all be wasted work.

Laziness Isn’t Always Bad

One of the principles of LINQ is to be lazy. A LINQ query won’t do any work unless you force the query to do the work. Even when a query does perform work – it does the least amount of work possible. If you wanted a List<Book> as a result, you’d have to force LINQ to create the list:

List<Book> filteredBooks =
               books.Where(book => book.Title.StartsWith("R"))
                    .ToList();

Instead of lists, LINQ works with a beautifully pure abstraction called IEnumerable<T>. It’s defined like so:

public interface IEnumerable<T> : IEnumerable
{
    IEnumerator<T> GetEnumerator();
}

The only thing you can do with an IEnumerable<T> is ask for an enumerator. An enumerator is something that knows how to visit each item in a collection. Some languages call these enumerator things “iterators”, because they iterate over a collection of objects, returning each object and moving to the next.

The in-memory LINQ operators, like Where, OrderBy, and Select, all work on inputs that implement IEnumerable<T>. That means the operators work on anything that can be enumerated over in a one-by-one fashion. The beauty is that there are so many data sources that these operators can work on, because IEnumerable<T> has such simple demands. Arrays are enumerable, lists are enumerable, dictionaries, trees, stacks, queues, files in a directory, elements in an XML document - all enumerable. Even a simple string is enumerable, since it is composed from a sequence of individual characters.

These same operators return IEnumerable<T>, because it’s the lowest common denominator for everything you’d ever need from a query. Plus, it’s lazy. You want to count the results? The Count operator will get the enumerator and move through the results, one by one, to sum the total number of items. You want to make a concrete list from the results? The ToList operator will get the enumerator and move through the results, one by one, adding each item to a new list it creates. Do you want just the first item in the results? Then the enumerator does just a little bit of work to find that first item. In most cases it does not need to iterate the entire collection to find the first item. Enumerators are lazy, too.

The important point is that the enumerator itself doesn’t perform any useful work. It’s you, or the other LINQ operators, that use the enumerator to iterate through the result and produce something meaningful. In the odd case that you never need to look at the result - no enumeration work is performed at all. No lists area created. Pure laziness!

Summary

The beauty of IEnumerable<T> is that it only says “you can get something to enumerate this”. To return something that offers the possibility of enumeration is very little work. And no work is needed unless you actually count the results, create a list from the results, or bind the results to a control for display.

The interface IEnumerable<T> is so wonderfully lazy it inspired me to write a short, short story. If you came to read about LINQ, skip the story as the words are entirely uninteresting and mostly devoid of meaning. 

The Lazy Leopard and the List

The scientist approached the big cat with a notepad and a pencil in her hands. She was worried, of course. The cat was a predator, and likely to be hungry at this hour of the day. “I need to know”, she asked the cat, “do you keep a list of things to do each day?”

The cat stirred. He was a snow leopard with dark rosettes blotted onto his thick, cream colored fur. The big cat’s eyes were only half open, but he turned and focused them on her.

“I don’t keep a list, dear lady”, he said, followed by a rumbling yawn. “I keep an enumeration”.

“An enumeration?”, she asked.

“Yes, an enumeration”, he replied. “Lists are like the gold bracelet on your wrist, dear lady. Very tangible – very concrete things, lists are. Keeping a list of everything I might want to do is a burden and chore. I’d need to carry your paper, and your pencil”, he said, with his eyes focusing on her hands.

The scientist’s pencil raced across her notebook as she transcribed every word the leopard spoke. She glanced at him as he began to stare, and instinctively pulled herself a little further away.

The leopard continued. “With a list I’d have to add things, and remove things, and constantly reorder the things I want to do. Too much work”, he said, shaking his head. “Do you know what I can do with an enumeration?”, he asked.

She paused at the leopard’s question, and pushed her hair back  - she wore glasses when she worked. After some thought, she asked, “Enumerate it?”

“Yes, dear lady”, said the leopard. “I can enumerate it. I enumerate the possibilities one by one, and find the perfect fit for this moment in my life. If I’m thirsty, I’ll find water. If I’m sleepy, I’ll find a place to sleep.” He tilted his head slightly to the right. “If I’m hungry, I’ll find food”, he said.

She finished writing the leopard’s last words and glanced up. Was that a tooth showing? Was he hungry now?

The cat started speaking again.

“One of the wonderful things about enumerations is they theoretically last forever. Lists have a beginning and an end – an Omega for every Alpha. With enumerations, you can keep asking for the next thing, over and over and over again. I ask for them when I’m ready to do something. If I’m tired of doing, they’ll still be there tomorrow. You might say it’s unpredictable behavior, I say I’m just being lazy. Either way, I can’t help it, it’s in my genes”. His soft voice trailed off with a tired tone.

“You intended to live forever?”, she asked. The leopard snarled. Or smiled. She couldn’t quite tell.

“No, dear lady”, he said. “I said the enumeration theoretically lasts forever. One day I’m sure my enumeration will run out of things to give me, or maybe I’ll just be too tired to ask for the next thing, so I’ll sleep forever. I don’t know how it ends. Maybe I should ask you.”

She looked at him again. She felt uneasy now, being here with a leopard. He seemed nice enough, as leopards go, and he certainly gave her interesting topics for research, but he was still a leopard. A carnivore. He was not a beast to be trifled with. She could never let her guard down again.

“I don’t know how it ends either”, she said, and closed her notepad. She tucked her pencil behind her ear, backed away from the cage, and left the leopard alone with his enumeration.

Steve Smith has posted a couple of interesting probability problem solvers on his blog recently. His most recent one is a twist on the Monty Hall problem. I mentioned the Monty Hall problem and its solution in an earlier blog entry of mine, For Some Probability Problems, Seeing Can Be Believing:

When facing a particularly tough probabilty problem sometimes it helps to sit down and draw out the possible outcomes for the given scenarios. In doing so, things that may otherwise seem very complex boil down to a very apparent solution or explanation.

A great example of simplicity through diagraming is the Monty Hall problem, which goes like this: You are a contestant on the TV game show, Let's Make a Deal and are presented with three doors. Behind one door there is a luxury car. Behind the other two doors there is a poster of a car. The game show host asks you to pick a door. After choosing one, the game show host reveals one of the other doors that has a poster behind it and then asks you if you want to change your door choice to the other unopened door. What is the best strategy?”

The answer is that you should change the door you had selected, because there's a 2/3 probability that the door you did not select houses the luxury car. In other words, there is a 2/3 probability that the door you picked has the remaining poster. The solution seems backwards. It seems paradoxical that the door the host reveals to have a poster of a car could have any influence on your initial selection. But if you sit down and draw out the various probabilities you can see that there's there are more good outcomes if you make the switch than if you hold fast to your initial choice. The following diagram was created by Rick Block; here, instead of a poster of a car you get a goat. Same difference.

Steve's most recent puzzle asks what would happen if instead of a single contestent in the Monty Hall problem there were multiple participants:

I take 3 plain envelopes and put a $100 bill inside one of them, seal them, and give one to you, one to Bob, and one to Carrie, at random.  Then I randomly ask one of you to open an envelope - for the sake of argument let's say I choose Carrie.  Carrie opens her envelope to reveal that it is empty.  Now I offer you the choice to trade envelopes with Bob - should you trade? And now I ask Bob the exact same question.  Should he trade?

As written, Steve's problem isn't that interesting. If he is randomly choosing one envelope to open then there is no advantage to be gained by changing or not changing your envelope. Now, if Steve were to reword the problem as follows (as I believe was his intention), then things are a bit more interesting:

I take 3 plain envelopes and put a $100 bill inside one of them. I then let you choose one, and then give the other two to Bob and Carrie.  I then open one of the envelopes (Carrie's or Bob's) knowing that it is empty. Now I offer you the choice to trade envelopes with Bob - should you trade? And now I ask Bob the exact same question.  Should he trade?

There is a big difference between these two questions. In the first one, everything is random; in the second one, the host (Steve Smith) knowingly opens one of the envelopes that is empty. It's also important that the host is forced to open Bob or Carrie's envelope, and cannot attempt to open yours.

The answer to this problem can be worked out the same way as the traditional Monty Hall problem - by putting pen to paper and drawing out the probability tree, although I used the name Cal instead of Carrie. (See the full size image...) 

As you can see, your odds are maximized if you get your hands on your contestant's envelope, while your opponent's best bet is to keep his envelope. Therefore, whoever gets to make the final change is the person with the best odds.

Another tool for determining how a probability puzzle works out is through empirical means. You can oftentimes write a terse program that simulates the problem at hand thousands of times and tallies up the various outcomes. I created such a program in C#.

I set ITERATIONS to 50,000 and let 'er rip. The following output confirmed my initial assumptions:

Percentage of Wins When Swapping and Contestant Swapping: 0.08266
Percentage of Wins When Swapping and Contestant Stays: 0.16618
Percentage of Wins When Staying and Contestant Swapping: 0.16482
Percentage of Wins When Staying and Contestant Stays: 0.08318

Percentage of Contestant Wins When Swapping and Contestant Swapping: 0.16754
Percentage of Contestant Wins When Swapping and Contestant Stays: 0.08386
Percentage of Contestant Wins When Staying and Contestant Swapping: 0.08262
Percentage of Contestant Wins When Staying and Contestant Stays: 0.16914

As you can see, your chances of winning are best when you swap and your contestant stays or when you stay and your contestant swaps - namely, when you get your hands on your contestant's envelope. Your contestant, however, does best when he hangs onto his envelope, either by having you both stay or both change.

Every couple of weeks or so I seem to have a need to transfer (or receive) a relatively large file to another colleague or client. It might be a database zipped up into a 50 MB file, artwork totalling hundreds of megabytes, or various DLLs and source code. Over the course of my career I have used the following techniques for transferring data:

• E-mail - the most straightforward technique for sharing data. The major con is that many email providers reject attachments over a certain threshold, usually in the 2-10 MB range. Also, e-mail is inherently insecure, making it a non-ideal medium for transferring sensitive data like passwords, credit card numbers, social security numbers, and so on.
• Upload/Download from a Website - if you have write access to a website you can always upload the data to share and then send a URL to the file(s). The downside here is that you need a website. Security also requires a few extra steps - you need to have some mechanism to only allow the intended recepient from downloading the article.
  
  Side story: Back in 2002 Microsoft released Web Matrix, which later became Visual Web Developer Express Edition. Prior to announcing the release Microsoft and asp.netPRO Magazine asked me to write an overview of the tool to be included in the asp.netPRO edition that came out right when the product was released officially. There were a group of other ASP.NET folks at the time who also knew about this project and were invited to review my article. I distributed it by putting the article up on a website and provided this group a link to download the article. A couple days later I left for a short vacation during which I did not check e-mail or phone messages.
  
  The day I left town one of the reviewers leaked the article to Neowin.net, which linked directly to the URL on my website. Shortly after the link was published on Neowin.net I started getting e-mails and phone calls from Scott Guthrie asking me to take down the link. Of course, I did not get these messages until a few days later upon return from my vacation.
  
  I never did find out who leaked the link/article. I guess it didn't really matter in the end, and it wasn't the article on my server that caused the leak as someone else sent in the link to Neowin.net and they could have just as easily sent the Microsoft Word document. But still, I was rather red-faced when having to explain to the guys at Microsoft why the article was accessible to anyone on the Net and why it took so long to get the article down once it had been compromised.
  
• Use a Third-Party Service - there are a plethora of web-based services that are built to allow large files to be transferred between two individuals. Most work by you uploading a file to a site, where it is stored temporarily. You then e-mail the recipient a link to download the attachment. I have used the free service TransferBigFiles.com many times and heartily recommend them. You could also use DropBox and place the file to share in the Public folder. The benefit of this approach is that you don't need a website; however, the security concerns remain. In fact, you might consider the security concerns even (slightly) greater because you are now adding an intermediary into the equation (the folks at TransferBigFiles.com or at DropBox).
• Physically Deliver the Data - the most secure option is the physically deliver the data - put it on a thumb drive or burn it to a DVD and then mail or hand deliver the data. This approach takes the most work, but is the most secure, especially if you hand deliver the data. And if you are transferring extremely large amounts of data - terrabytes, let's say - this approach would likely be the fastest.
  
  I used this approach 8 years ago when transferring large amounts of information between myself and a publisher. At the time I only had dial-up access. When the publisher needed to send me several hundred megabytes worth of materials - mostly artwork from previous editions - it was more efficient to mail me a CD. I also had a student in one of my Web Services class, must have been four years ago or so, whose company was running computer simulations that would output several GBs worth of data per test run. To send the day's test data to another facility they would plug in an external hard drive, copy over the data, and then use a mail carrier to overnight deliver the drive. I wonder what techniques they use today.

How do you share large files with others?

Since NHaml has broken out of MvcContrib (which is a great decision I think, I hate contrib projects in general since they are very hard to maintain) there isn't a great place to ask questions about using NHaml so I went ahead and created a NHaml-Users Google group.

-James

I try to keep this blog tightly focused on technical issues, but I wanted to share this happy bit of news: on September 1st my wife gave birth to our daughter, Alice Mitchell. Mom & baby are healthy and happy.

If you'd care to read a bit more about this, I've blogged about it on my personal blog, ScottOnLife.com: Then There Were Three.