Posts tagged General technology

A Relative Path Facility For Castle Windsor

At work, we use Castle Windsor for Dependency Injection. In Castle Windsor, as with any dependency injection framework, you can configure components identified by an interface, that can be resolved at runtime using the dependency injection framework. Components can have dependencies, which can be yet other components and so on. In this way, you can have your dependency injection framework create a whole graph of objects for you.

One limitation we run into now and then, is with components, that depend on a file path to work. Typically, we need to know the full path of the file to load it. But hardcoding the full path in the configuration file is generally a bad idea, it will create problems when you move your web application between environments. Also, we cannot just pass the path as a virtual path to the component and then have the component call Server.MapPath to map the path - since that would mean changing the interface of the component just to accomodate the injection framework, which is not a good idea. And, what is worse, you would create a dependency on System.Web in a place where it probably isn't needed.

Now, one way to get around this would be to create a wrapper interface, IFilePath, which only should exist in order to be passed into the component and being able to convert the path. This also involves changing the component and generally feels like a bad idea.

Luckily, the Windsor IoC container offers a large variety of extension points - one being facilities. So I wrote a facility, that allows paths configured in Castle Windsor to be relative. The way this works is by registering an ISubDependencyResolver in the IKernel instance. When resolving a dependency, Windsor will ask the ISubDependencyResolver whether it can resolve the dependency using the CanResolve method. By examining the passed ComponentModel and in particular it's configuration node, I look for a custom attribute on the dependency, pathType. If found (and the dependency is of type string), then we can easily resolve the dependency by taking the relative path in the configuration tag and making it absolute.

This will allow you to have your Windsor configuration look like this (notice the one-line facility registration - this is what registers the custom facility in Windsor, and makes us able to register the path dependency as a virtual path):

         1:   <castle>
         2:     <facilities>
         3:       <facility id="pathResolver" type="dr.Castle.WebPathFacility.RelativePathSupportFacility, dr.Castle.WebPathFacility" />
         4:     </facilities>
         5:     <components>
         6:       <component id="dummy"
         7:                  service="dr.Castle.WebPathFacility.Test.IDummy, dr.Castle.WebPathFacility.Test"
         8:                  type="dr.Castle.WebPathFacility.Test.Dummy, dr.Castle.WebPathFacility.Test" >
         9:         <parameters>
         10:           <path pathType="Relative">App_Data/test.xml</path>
         11:         </parameters>
         12:       </component>
         13:     </components>
         14:   </castle>
 

The valid values for pathType are:

         1:         private enum PathType
         2:         {
         3:             /// <summary>
         4:             /// The path is absolute (we will do nothing to it).
         5:             /// </summary>
         6:             Absolute = 0,
         7:             /// <summary>
         8:             /// The path is a virtual path to a web application resource.
         9:             /// </summary>
         10:             Virtual,
         11:             /// <summary>
         12:             /// The path is relative to the current directory.
         13:             /// </summary>
         14:             Relative
         15:         }

The code for the facility it self is really simple, since it simply registers our dependency resolver to the Kernel. The advantage of using a facility, is that it can be declared in the config, and Windsor will automatically initialize for all containers you create:

         1: 
        using Castle.MicroKernel.Facilities;
         2:  
         3: 
        namespace dr.Castle.WebPathFacility
         4: {
         5:     public class RelativePathSupportFacility : AbstractFacility
         6:     {
         7:         protected override void Init()
         8:         {
         9:             Kernel.Resolver.AddSubResolver(new PathParameterDependencyResolver());            
         10:         }
         11:     }
         12: }

Finally, the implementation of ISubDependencyResolver, that makes this possible:

         1: 
        using System;
         2: 
        using System.Collections.Generic;
         3: 
        using System.IO;
         4: 
        using System.Linq;
         5: 
        using System.Web;
         6: 
        using Castle.Core;
         7: 
        using Castle.MicroKernel;
         8:  
         9: 
        namespace dr.Castle.WebPathFacility
         10: {
         11:     /// <summary>
         12:     /// Custom dependency resolver, that will inspect the parameters collection for the pathType attribute, and, if found, convert the dependency to 
         13:     /// a absolute path based on the path type.
         14:     /// </summary>
         15:     class PathParameterDependencyResolver : ISubDependencyResolver
         16:     {
         17:         /// <summary>
         18:         /// Holds the supported conversion operations.
         19:         /// </summary>
         20:         private static readonly Dictionary<PathType,Func<string, string>> conversions = new Dictionary<PathType, Func<string, string>>
         21:                                                                                            {
         22:                                                                                                {PathType.Absolute, path => path},
         23:                                                                                                {PathType.Relative, path => Path.Combine(Environment.CurrentDirectory,path) },
         24:                                                                                                {PathType.Virtual,  path => HttpContext.Current.Server.MapPath(path)}
         25:                                                                                            };
         26:  
         27:         /// <summary>
         28:         /// Cache of the type path parameters.
         29:         /// </summary>
         30:         private readonly Dictionary<string,PathParameter> typePathParameters = new Dictionary<string, PathParameter>();
         31:  
         32:         /// <summary>
         33:         /// Resolves the specified dependency.
         34:         /// </summary>
         35:         /// <param name="context">Creation context</param>
         36:         /// <param name="contextHandlerResolver">Parent resolver</param>
         37:         /// <param name="model">Model of the component that is requesting the dependency</param>
         38:         /// <param name="dependency">The dependcy to satisfy</param>
         39:         /// <returns><c>true</c> if the dependency can be satsfied by this resolver, else <c>false</c>.</returns>
         40:         /// <returns>The resolved dependency</returns>
         41:         public object Resolve(CreationContext context, ISubDependencyResolver contextHandlerResolver, ComponentModel model, DependencyModel dependency)
         42:         {
         43:             PathParameter parameter = GetPathParameter(model, dependency);
         44:             if (parameter == null) 
         45:                 throw new ApplicationException(String.Format("Cannot resolve dependency {0}", dependency));
         46:             if (!conversions.ContainsKey(parameter.Type))
         47:                 return parameter.Value;     // Unknown conversion
         48:  
         49:             return conversions[parameter.Type](parameter.Value);
         50:         }
         51:         /// <summary>
         52:         /// Determines whether this sub dependency resolver can resolve the specified dependency.
         53:         /// </summary>
         54:         /// <param name="context">Creation context</param>
         55:         /// <param name="contextHandlerResolver">Parent resolver</param>
         56:         /// <param name="model">Model of the component that is requesting the dependency</param>
         57:         /// <param name="dependency">The dependcy to satisfy</param>
         58:         /// <returns><c>true</c> if the dependency can be satsfied by this resolver, else <c>false</c>.</returns>
         59:         public bool CanResolve(CreationContext context, ISubDependencyResolver contextHandlerResolver, ComponentModel model, DependencyModel dependency)
         60:         {            
         61:             if ( dependency.DependencyType == DependencyType.Parameter && dependency.TargetType.Equals(typeof(string)) )
         62:             {
         63:                 PathParameter parameter = GetPathParameter(model, dependency);
         64:                 return parameter != null;
         65:             }
         66:             return false;
         67:         }
         68:  
         69:         /// <summary>
         70:         /// Finds the parameter by looking at the cache, then in the model configuration.
         71:         /// </summary>
         72:         /// <param name="model"></param>
         73:         /// <param name="dependency"></param>
         74:         /// <returns></returns>
         75:         private PathParameter GetPathParameter(ComponentModel model, DependencyModel dependency)
         76:         {
         77:             if (!typePathParameters.ContainsKey(model.Name))
         78:                 typePathParameters.Add(model.Name, GetPathParameterInternal(model, dependency));
         79:  
         80:             return typePathParameters[model.Name];
         81:         }
         82:  
         83:         /// <summary>
         84:         /// Finds the parameter by looking at the model configuration.
         85:         /// </summary>
         86:         /// <param name="model"></param>
         87:         /// <param name="dependency"></param>
         88:         /// <returns></returns>
         89:         private PathParameter GetPathParameterInternal(ComponentModel model, DependencyModel dependency)
         90:         {
         91:             var parametersContainer = model.Configuration.Children.SingleOrDefault(n => n.Name == "parameters");
         92:             if ( parametersContainer != null )
         93:             {
         94:                 var parameterNode = parametersContainer.Children.SingleOrDefault(n => n.Name == dependency.DependencyKey);
         95:                 string pathType = parameterNode.Attributes["pathType"];
         96:                 if (pathType != null)
         97:                 {
         98:                     PathType type;
         99:                     if (!Enum.TryParse(pathType, true, out type))
         100:                         throw new ApplicationException(
         101:                             String.Format("Configuration error: Invalid pathType value '{0}'", pathType));
         102:  
         103:                     return new PathParameter {Type = type, Value = parameterNode.Value};
         104:                 }
         105:             }
         106:             return null;
         107:         }
         108:  
         109:         /// <summary>
         110:         /// Holds a path parameter
         111:         /// </summary>
         112:         private class PathParameter
         113:         {
         114:             /// <summary>
         115:             /// Value as entered in config
         116:             /// </summary>
         117:             public string Value { get; set; }
         118:             /// <summary>
         119:             /// Type of path.
         120:             /// </summary>
         121:             public PathType Type { get; set;}
         122:         }
         123:  
         124:         /// <summary>
         125:         /// Defines the types of paths supported by <see cref="PathParameterDependencyResolver" />
         126:         /// </summary>
         127:         private enum PathType
         128:         {
         129:             /// <summary>
         130:             /// The path is absolute (we will do nothing to it).
         131:             /// </summary>
         132:             Absolute = 0,
         133:             /// <summary>
         134:             /// The path is a virtual path to a web application resource.
         135:             /// </summary>
         136:             Virtual,
         137:             /// <summary>
         138:             /// The path is relative to the current directory.
         139:             /// </summary>
         140:             Relative
         141:         }
         142:     }
         143: }

Now, I am finally able to use virtual paths in my configuration files, with a minimum of noise. Great. Please notice, that the "Relative" path type might not make sense for a real application (since it uses Environment.CurrentDirectory as base), but it can be really helpful in test configurations. The primary reason for creating this is pathType="virtual", which maps to Server.MapPath.


Getting your public IP from a PowerShell script

I often work on computers at different locations, and often, I need to know what public IP I am using to connect to the internet. Of course, this is easy to find out - I can just go to a website that tells me my IP, such as http://myip.dk/.

But I find this to be suboptimal. If I am configuring something, finding the IP involves firing up a browser, going to the site, and copying the IP displayed. It is a speed bump when I am trying to be productive - and the display-my-ip sites are often covered in commercials, which I dislike.

So today, I decided to write a PowerShell script that can tell me my current public IP address. First, I needed a reliable way of finding it. Of course, I could just screen-scrape off a site such as http://myip.dk/, but it has some disadvantages. I can't know if the html structure will change - and it would mean that I would have to download all of the HTML just to get a few bytes of IP address. Furthermore, I don't know whether it would be legal at all.

Therefore, I started by writing a small ASP .NET HTTP handler, that could tell me my IP. I put the following simple code in the ProcessRequest method:

 1:     public void ProcessRequest (HttpContext context) {
 2:         context.Response.ContentType = "text/plain";
 3:         context.Response.AddHeader("X-RemoteIP", HttpContext.Current.Request.UserHostAddress);
 4:         context.Response.Write(HttpContext.Current.Request.UserHostAddress);        
 5:     }

This simply writes the IP address the handler is accessed with, to the response as well as to a custom http header. I then deployed this handler to my website.

Next, writing the PowerShell script, was equally simple; we can simply use the handy System.Net.WebClient class:

 1: $ipFinderHost = "http://www.somedomain.org/GetIP.ashx"
 2: $c = new-object System.Net.WebClient
 3: $c.DownloadString($ipFinderHost)

And voila, I have a PowerShell script that displays my public IP address. And, since I have the PowerShell Community Extensions installed, I can use the set-clipboard cmdlet to copy it to the clipboard.

 1: get-myip | set-clipboard

Much nicer than manually copying from the text in a browser :-) If you decide to use this script yourself, obviously you will need to change the URL in the script to where you have deployed the GetIP.ashx handler.


Introducing: The Google Chrome Password Recovery Tool

Today, I wanted to backup all my passwords stored in Google Chrome. I thought that would be an easy task, but it turns out, that this is not supported, at least as far as I can tell. There is an option to view the shared passwords one-by-one, but that was not really an option for me.

So, I decided to write a small program to extract the passwords from Chrome. Since Chrome (or Chromium, to be exact), is open source, I pulled the source from http://dev.chromium.org/getting-involved, compiled it, and starting looking around trying to figure out how passwords are stored. The setup and build experience was much nicer than what I have tried with other open source projects I have looked at; there are detailed build instructions with only a few steps available, and after setting up, it just works, in a Visual Studio 2008 solution. A full recompile does take some time however (45 minutes on my machine).

I quickly found out that Chrome stores most of its user and configuration data in small SQLite databases stored on disk in the AppData/Local/Google/Chrome/User Data directory. So, reading the data was no problem after grabbing a copy of the ADO .NET Provider for SQLite, as well as the sqlite3.dll binary from http://www.sqlite.org/. The data I was after (user names and passwords) is stored in the file named Web Data. This contains a table named logins, which contains the URL for which the login is valid, some details about the html form where the password has been used (in order to allow Chrome to auto-fill password boxes for you), and the username and password. It also contains a "preferred" and "blacklisted_by_user" column.

Decrypting the passwords

The passwords is, obviously for security reasons, not stored in plain text. Rather, they are encrypted, so I needed to figure out how they are encrypted and how to decrypt them. The answer lies in the Chromium source, where the Encryptor class contains the following method:

bool Encryptor::EncryptString(const std::string& plaintext,
                              std::string* ciphertext) {
  DATA_BLOB input;
  input.pbData = const_cast<BYTE*>(
    reinterpret_cast<const BYTE*>(plaintext.data()));
  input.cbData = static_cast<DWORD>(plaintext.length());

  DATA_BLOB output;
  BOOL result = CryptProtectData(&input, L"", NULL, NULL, NULL,
                                 0, &output);
  if (!result)
    return false;

  // this does a copy
  ciphertext->assign(reinterpret_cast<std::string::value_type*>(output.pbData),
                     output.cbData);

  LocalFree(output.pbData);
  return true;
}

As it turns out, the Windows Data Protection (DPAPI) is used to encrypt the data, namely the CryptProtectData function as shown above. Therefore, I can relatively easy decrypt the data, using the CryptUnprotectData function, as long as I do not try to decrypt the password of other users - the DPAPI encrypts with a key based on the current user's login credentials. I first tried to do the decrypting in C# using P/Invoke the the CryptUnprotect function, but for some reason, I could not get that to work. I kept getting a credentials dialog from Windows when trying it, which is not what i want. Luckily, after googling a bit, I found out that there already exist a managed wrapper for doing this, namely the ProtectedData class. After switching to using this, there were no problems decrypting the passwords.

The password recovery tool

I wrote a tiny C# console program to dump the passwords. They can be dumped to the console (default) or to an XML file. Running the program without parameters will try to dump all the passwords to the console. You might need to have Chrome closed while doing this. Run the program with the -help switch to display a short usage information message.

I am including the program for download here - both in a precompiled form and the C# source. It requires .NET Framework 3.5. The program as well as it's source is free for use non-commercially and is provided without any warranty or obligations for me, neither explicit or implied. It probably won't kill your cat, but don't come to me crying about it if it does ;-). If you wish to use the source or derivate thereof in a commercial product, contact me for permission first.

Download:

What is missing ?
It would be nice to have an option to import the exported passwords into a new Chrome installation on another computer. I am considering adding it, but don't really need it at this time. It should be relatively easy - if you happen to develop the feature based on my work, please email me the source.


Thoughts on Google Chrome

I admit it, I am a fan of the new Google Chrome browser.

This made me chuckle, however:
Google vs Microsoft ... ?

Is Google preparing to dominate the world ?


A Stopwatch for Windows Mobile

I have got a new mobile phone, a HTC Touch Diamond. Besides the fact that it has a sleek design and is much easier to work with when reading email and browsing the web than my old phone.

However, that it is not the only reason for buying the Diamond. Another, very important reason, is that it runs Windows Mobile 6.1 - and therefore I can write my own programs for it using pretty much the same toolset as I use for any other .NET program. Granted, there are stuff missing in the Compact Framework compared to the full-blown framework (Expression trees anyone ?), but it is normally quite easy to find alternatives, and the Compact Framework does make it quite easy to program the device.

My first application for the Touch is a simple Stopwatch program. I wrote it, because there was no stopwatch and/or timer program on the Touch when I got it, so why not write my own ;-) The application it is quite simple, but I learned quite a deal about the device and the Compact Framework while developing it. It essentially relies on the Environment.TickCount counter to measure time, so it might not be 100% accurate - but for my needs (such as heating pizza's), it is quite sufficient.

If anyone's interested, you may download the source from here. If you want to compile it, you will need a copy of the OpenNET CF Framework, because I needed to use some parts of it for making the phone vibrate when the alarm goes off. (It could be replaced with some P/Invoke calls, but i got lazy ;-)

The application itself has the following features:

  • Simple stopwatch
  • Timer with alert (vibration and sound)
  • Configurable alarm sound (only .wav files, sorry).
  • Settings are remembered (stored in Application Data)

Last Day at Jaoo

Wednesday was the last official day of the JAOO conference, and once again it featured a bunch of interesting talks. I attended these:

50 in 50
This was todays keynote by Richard P. Gabriel and Guy L. Steele Jr. Before the talk, there had been some speculations about the title; was it 50 programming languages in 50 minutes ? Or what did it mean exactly ? It turned out to be 50 comments about programming and programming languages, in 50 minutes. These focused on the history of programming and did so in an entertaining and enlightening manner. This was a certainly a great talk - and for a "young" programmer like my self what was not even born when Ada and Algol 60 appeared; it provided also some historical insight. Only downside to this talk, is that the schedule was affected by the fact that it was more like 50 in 75 - the talk took about 75 minutes; but with this quality of technical and on the same time entertaining talk, that does not really matter for me.

Five Considerations For Software Developers
This was also a dual talk with two presenters - Frank Bushmann and Kevlin Henney. They talked about architecure and specifically five considerations that drives design quality. Those were:
  • Economy - the idea that software must be built for a reason and should not have an complicated or elaborate design just because it *might* be needed in the future.
  • Visibility - in the sense that the design must be easily discoverable.
  • Spacing - basically the idea to separate concerns and make sure not to bake the design into deep inheritance hierarchies that xould better be expressed with composition.
  • Symmetry - in that API's should be symmetric with the example that if you can create something with a Factory, said Factory should also be able to destroy it
  • Emergence


LINQ + New Microsoft Things
This talks title is actually wrong, since Erik Meijer primarily talked about LINQ, and very little about "New Microsoft Things". To be fair, he did not have much time to cover it all since the talk got started late because of the schedule slip at the keynote earlier on the day. LINQ was covered well, however, and from a slightly different angle than Anders Hejlsberg talked about earlier in the week. Erik talked about Expression trees and how they represent code as data. This makes it possible to hand an expression tree to an interpreter for a given query language, that can then execute it in the given domain. This is why we (in theory) could forget all other query languages such as XQuery or SQL, and only use Linq-to-Xxx - given that someone writes a Xxx extension to LINQ, of course.

Real World Refactoring
This talk about Refactoring by Neal Ford addressed the challenges that goes into actually performing refactorings in code. It was very hands-on and offered some good advice on how to structure refactorings. One of the best pieces of advice, I think, was to time-restrain major (multi-day) refactoring efforts to an estimated period of time before-hand. If you cannot complete the planned refactoring in the planned time, take the time to rethink the problem, and find out if you are doing it right. If not, you can throw the refactored code away and try again, instead of keeping on a track, that might resolve to more complicated code than before; because new knowledge has beeng gained during the process or because the refactorings was not well enough planned and thought out in advance.

JavaScript As An Assembly Language
This second presentation by Erik Meijer was primarily about Volta, an exciting new technology from Microsoft's Live Labs. The project basically promises to make it easier to make multi-tier applications that can run on the server and work with any client, with parts being executed on the client. This is done by decorating methods with custom attributes, that marks them for running on the client. The Volta compiler will then "compile" those to javascript, that can run on any client (or, if Silverlight is available on the client, the code will run in Silverlight as .NET IL). Erik explained the technology behind, and how they generate javascript code and the various problems involved in that. I do not think that this technology is quite ready to be used in the wild yet, but it should definitely be interesting to see how it evolves in the future. The documentation site on Live Labs seems to be down for the moment, however, this blogpost also explains the technology in more detail.

Concurrent Programming with Concurrent Extensions to .NET
In this talk, Joe Duffy, gave an introduction to the parallel extensions to .NET, a new API for writing concurrent applications with .NET. These extension is in CTP right now (so it's preview technology, not recommandable for production use). Joe promised though, that these APIs will be part of the .NET Framework version 4 release. These new APIs promise to make it easier to write concurrent applications with .NET with little overhad, both mentally for the programmer, but also performance-wise for the machine. The presentation featured running demos and code, and I believe that the new APIs are quite well-designed and that there is definitely a need for this kind of API in todays world of multi-core hardware. However, as Joe pointed out, there is no such thing as a free lunch; and even when using this API, of course you need to think hard over concurrency issues and side-effects before you can put it to use. The system makes it easier for you to program concurrently; but you can still fail badly if you do not understand what it does under the covers.

JAOO Day Two

Today on JAOO has also been packed with interesting talks. I attended these:

V8: The Javascript engine inside Google Chrome
Keynote by Lars Bak about the all-new javascript engine that was implemented by his team in Århus for Google Chrome. This javascript engine is about 10 times faster than other javascript implementations. Lars explained how this is possible in a very dynamic language like javascript, by creating classes that can be reused; if another object with the same properties is created, which tends to happen often. Optmizations can then be applied to those classes. Furthermore, Chrome compiles the Javascript code to native code so that it can run really fast on any platform. Lars mentioned a few other major improvements, and this was a very inspiring and interesting talk - even though he had some problems with the projector in the beginning.

Failure Comes In Flavours
This talk by Mikael Nygard (who is not danish, by the way, though his name resembles a danish name pretty much) was divided into two sessions. In the first, Mikael talk about anti-patterns that lead to failure - such as depending too hard on third parties, or waiting forever for some external call. This talk featured some "war stories" about failures Mikael had helped to resolve in the past. In the second session, Mikael offered his advice on how to avoid failure and some patterns that can help in this. Though focus was very much on big enterprise SOA systems; the principles can be applied anywhere. I learned a thing or two in this session, that can be applied to my every-day work on web applications in a smaller scale.

Not your Grandfather's Architecture: Taking Architecture into the Agile World
In this talk, James Coplien talked about Agile architecture. He presented his ideas about adding roles as a concept to the object-oriented world of classes and objects. He argued that programs could be made simpler and get rid of polymorphism using his approach; I am not sure that I agree. There was no working code demo, so his ideas is still kind of abstract in my mind.

Successfully applying REST - Integration, Web-style
Stefan Tilkov talked about REST integration; a subject that I find very interesting, since we already use it in projects at work, and are planning to use it even more extensively. Stefan had some very interesting points about REST, and how it can be applied to a SOA world. I really like the clean interface you can make of a REST service, without much of the overhead and scaffolding that is neccessary in SOAP, for instance.

Top Ten Software Architecture Mistakes
This was a talk focusing on what not to do in architecture; so that we can avoid mistakes or bad decisions in our architecture. The talker, Eion Woods, had his list of 10 mistakes and how to avoid them, which was presented with a little bit of humour. At least some of the items is going to be on my list of things to check before beginning development of new projects.


JAOO Day One

Today, I've been attending the first day at the JAOO Conference. It has been an interesting day, and I am amazed by all the very talented people that speaks here at the conference, as well as by the quality of the talks. I have been attending these sessions:

Opening Keynote: Where Are Programming Languages Going, by Anders Hejlsberg
In this talk, Anders presented his take on how the programming languages will be evolving in the future. One point made, was that programming languages has not evolved much over the last 25 years - which was exemplified by a "Hello world" program in Pascal versus one in C#. The C# one was neither the shortest nor the most concise one. Anders believes that we will se more implementation of what he calls "internal domain-specific-languages", such as LINQ. Furthermore, he thinks that in the future the gap between functional and imperative programming languages will diminish, since they are already today starting to borrow the best elements from each other. Finally, he talked about concurrency and in-language concurrency constructs such as isolation, which Anders predicts also will be part of the main-stream languages in the near future.

Continuous Integration
The Continuous Integration talk was a great introduction to me into Continuous integration, since it is something, that I have little to zero experience with. We do have a build server that does nightly builds at work, but as Chris Read, the speaker, pointed out, that is not CI, though it is a step on the way toward succesful CI.

Google Chrome: The Invisible Browser
This was a talk by Ben Goodger, the Google Chrome UI tech lead, about the minimalist Chrome UI and the architecture and decisions behind it. It was interesting, though there was not much new to learn here. (Or perhaps I should know better than to attend UI talks, when UI does not really interest me. I am trying to learn ;-) )

LING and C# 3.0
This was the second talk by Anders Hejlsberg this day, and featured the new features in C# 3.0. This was info that I knew (mostly) in advance, but Anders explained both the how and the why behind the features - which was really interesting.

PowerShell
The talk about PowerShell was really good; even for me who know and use PowerShell in advance. It got beyond the covers on why the architecture and implementation works as it does, which was interesting and enlightening, and I left with a better understanding of PowerShell.

The Scala Programming Language
Scala is a language for the JVM, that I did not know much about in advance. The talk was interesting, but in "real life", I am probably never going to use it.

Why Functional Programming (still) Matters
This talk by Erik Meijer was propably the most interesting and entertaining one on Day One of JAOO. With enthusiasm, Erik explained about side-effects and why they are bad and what one should do about them (make them explicit if they cannot be avoided). He also demonstrated a few side effects, that can hit you in C# or other main stream languages with closures and lambdas, which was a pleasant reminder for me.


Attending JAOO

I will be attending the JAOO conference the next couple of days. It is my first time, but I have high expectations. I am not sure which sessions I will attend, but of course the opening keynote by Anders Hejlsberg will be a must.

I find it hard to choose - there are so many interesting subjects and speakers; and so little time ;-) If you are going to JAOO, feel free to drop a comment with suggestions on which sessions is a must for you, and why.

Parsing XML with PowerShell

I'm addicted to PowerShell. This cool scripting environment is simple to use, and with very few lines of script; it is possible to accomplish tasks that otherwise often would be a lot of tedious work. (If we didn't have PowerShell, I would propably wip up a C# program to do the same, but PowerShell is really lightweight, is interactive and is generally very forgiving for small tasks where you just "want the job done".

As an example, today I needed to look at a log files generated by Visual Studio to figure out why the environment wouldn't start on my home PC. As it turns out, these log files are actually XML files. Of course I could have just started reading through the XML, but all the angle brackets confuses my brain; when I'm actually mostly interested in the text content of the log file.

So, five minutes later, this 3-line script; parse-vslog.ps1 was born:

1: param( [string]$file = $(throw "required parameter" ) )
2: $log = [xml](get-content $file)
3: $log.activity.entry | select record,type,description | format-table -wrap -auto

This is what happens in the script:

On line 1, we declare that we need a $file parameter (variables and parameters is prefixed with $ in PowerShell), that should be required.

On line 2 we use the get-content cmdlet to get the contents of a file. PowerShell has a lot of XML helping features, one of which is the ability to "cast" the content to XML using the [xml] construct. What really happens behind the scenes, is that PowerShell instantiates an XmlDocument and loads the text content of the file in that.

Last, on line 3, we take advantage of the fact that PowerShell let's us select XML nodes by using simple dotted notation. Here we are interested in all the the /activity/entry nodes. We pass the result along the pipeline and selects the 3 most important values using the select cmdlet. And, lastly, we format the output nicely with format-table, specifying that we would like the cmdlet to auto-select the column widths (-auto) and that text output should be wrapped on multiple lines (-wrap).

So insted of having to look at XML that goes on like this:

1: xml-stylesheet type="text/xsl" href="ActivityLog.xsl"?>
2: activity>
3:   entry>
4:     record>1record>
5:     time>2008/06/15 15:44:18.220time>
6:     type>Informationtype>
7:     source>Microsoft Visual Studiosource>
8:     description>Visual Studio Version: 9.0.21022.8description>
9:   entry>
10:   entry>
11:     record>2record>
12:     time>2008/06/15 15:44:18.221time>
13:     type>Informationtype>
14:     source>Microsoft Visual Studiosource>
15:     description>Running in User Groups: Administrators Usersdescription>
16:   entry>
17:   entry>
18:     record>3record>
19:     time>2008/06/15 15:44:18.221time>
20:     type>Informationtype>
21:     source>Microsoft Visual Studiosource>
22:     description>ProductID: 91904-270-0003722-60402description>
23:   entry>
24:   entry>
25:     record>19record>
26:     time>2008/06/15 15:44:19.094time>
27:     type>type>
28:     source>Microsoft Visual Studiosource>
29:     description>Destroying Main Windowdescription>
30:   entry>
31: activity>
32:  

Now, I can get this much nicer output in the console (note that the XML above has been shortened for the blog. It was actually around 150 lines):

record type        description
------ ----        -----------
1      Information Visual Studio Version: 9.0.21022.8
2      Information Running in User Groups: Administrators Users
3      Information ProductID: 91904-270-0003722-60402
4      Information Available Drive Space: C:\ drive has 42128211968 bytes; D:\ drive has 38531145728 bytes; E:\ drive h
                   as 127050969088 bytes; F:\ drive has 117087354880 bytes
5      Information Internet Explorer Version: 7.0.6001.18063
6      Information Microsoft Data Access Version: 6.0.6001.18000
7      Information .NET Framework Version: 2.0.50727.1434
8      Information MSXML Version: 6.20.1076.0
9      Information Loading UI library
10     Information Entering function CVsPackageInfo::HrInstantiatePackage
11     Information Begin package load [Visual Studio Source Control Integration Package]
12     Information Entering function CVsPackageInfo::HrInstantiatePackage
13     Information Begin package load [team foundation server provider stub package]
14     Information End package load [team foundation server provider stub package]
15     Information End package load [Visual Studio Source Control Integration Package]
16     Information Entering function VBDispatch::GetTypeLib
17     Information Entering function LoadDTETypeLib
18     Error       Leaving function LoadDTETypeLib
19                 Destroying Main Window
 

I think this is a good representative of the strength of PowerShell. Using only a few lines of script and a minimum of time, I created a reusable script, that will probaply save a lot of time in the future.


Built on Sitecore Xpress

As from today, this blog runs on Sitecore Xpress. I have spent the day setting it up.

So, what is Sitecore Xpress ? Sitecore Xpress is a personal edition of the professional Sitecore CMS product, which just recently was released and made free of charge for developers. This is really cool, since Sitecore is a very good, mature and professional CMS, that has loads of features and is built on .NET. It also has a lot of extension points, which makes it a realy nice development platform for large enterprise websites. The free Xpress edition has almost all the features of the full product, though it is restricred to one user and can only be used for personal, non-commercial sites (details here).

Since we use Sitecore a lot at work, and it is so developer-friendly, the decition to use Sitecore Xpress for my personal website, was really easy to make. It will make a good platform for the various features I would like to add to my web site in the feature.

A nice example on how easy use and setup Sitecore is, that although there are no built-in blogging application in Sitecore, it took me just a day to setup the entire website, including the blog and migration of content from the old platform. The blog is actually a tree of so-called Sitecore content Items. Items is the cornerstone of Sitecore. Using a few XSLT renderings, the blog page, archives and individual posts and comments can be displayed. No additional code was written to achieve this. The only code I have written for this website so far, is a small User Control ( < 30 lines of code ) that allows users to add comments to the articles.

I am probably going to be blogging a bit more about Sitecore and ASP .NET related subjects in the future. I also hope that this change of platform will increase my motivation for writing more posts, so the frequency of new posts should increase.


NEF File reader is on hold ...

I have not had very much time to blog nor work on my NEF file reader project for a while. It has been turbulent and busy times at work, lately.

This post is just to let you know, that the project is curently on hold. Perhaps I will resume it later, when I have more free time at hand. If anyone is interested in a peek at the source code as it looks now (incomplete and raw), please let me know; and I can mail it to you. There is not too much functionality, however. The largest obstacle in getting further with the project, is the need to decode the Huffman-encoded sample bit lengths embedded in the file - I have not resolved this yet.


NEF file reader update

Progress on the NEF / RAW file reader is slow, since i only work on it occasionally when I feel like coding in my spare time. However, I made some progress tonight, that I would like to update you about.

First, NEF files are really TIFF files. So I need to parse them as a TIFF file to get some useful information out of them. TIFF files consists of so-called IFD tags, which in parts contains metadata about the image as well as the actual image data. Currently I have developed a basic IFD tag parser, that parses all of the IFD tags in a NEF file.

Furthermore I have found, by looking at the IFD tags, that there are at least 3 embedded images in each NEF files. These are a small, low-quality 160x120 thumbnail represented as RGB data, a nearly full size JPG in low quality, as well as the actual NEF image data. While implementing the IFD parser, I had a lot of help from this TIFF FAQ, as well as from the official Adobe TIFF specification version 6.

Of course, the NEF image data is the interesting part. I have studied the file format, and made these conclusions; mostly based on what I can tell from the dcraw source code:

  • The NEF file consists of Width * Height samples.
  • The data is represented as a Color Filter Array, that is each sample represents the intensity of a single color (since this is what the camera shoots, as many other digital cameras);
  • therefore, I must interpolate 2 of the colors for each sample, to get a full-color, full-scale image.
  • Each sample is 12 bits.

But this is not all; it is not so simple as to just read 12 bits per sample. The data is compressed. As far as I can tell currently; the following is the compression scheme.

  • A µ-law type compression curve is used to limit the number of possible sample values to 634 (and consequently companding the 12 bit values to log2(634) = 9.4 bits of actual accuray; while preserving the 12 bits dynamic range).
  • The curve values are embedded in the file.
  • Samples are indices to values in the curve.

Samples are encoded (compressed) like this:

  • A simple predictor is used, so each value read is actually the difference between a previous sample value and the current. This keeps values stored in the file low and keeps differences between each value needed to be stored, low.
  • Each value is  stored as a run-length encoded length (in bits) of the value, and immediately thereafter, the actual value.
  • Run length encoding is accomplished by using a classic Huffman table.

So, now I need to implement the reading of the actual sample values and interpolate the values to come up with a full color image. After that, I probably need to implement the reading or approximation of white balance of the image, so that the program will be able to produce actually usable images.


Hobby project - Nikon NEF file reader

So, my new hobby is digital photography. I have bought a slightly used Nikon D50 camera, and has already taken a lot of pictures.

If you're into digital photography, you will know that most cameras can produce both JPG and RAW output, where the JPG images are compressed by the camera firmware before being written to the flash card. The RAW files are uncompressed data as it was sampled by the camera CMOS (typically).  Remembe, JPG is a lossy compression format, so detail will be lost if you just save the JPG's. Therefore, if you are serious about photography, you will want to shoot in RAW.

However, as RAW files are essentially untouched image data, a RAW file needs to be processed by some image processing software before it can be used. This can be a cumbersome process, loading up the camera manufacturers software and maybe Photoshop or another image processing utility, processing each image, perhaps adjusting colors and saving the file into a more common format.

Therefore I have decided to try and create a .NET program that does the conversion for me. For "version 1", it will probably only support the NEF file format that my Nikon produces (Nikon's proprietary RAW format), but perhaps I will add support for others along the way. This program will not be a replacement for professional photo processing software, but is meant to be an easy to use tool, that can be used when one simply needs to convert a NEF file to a JPEG or bitmap representation quickly.

The NEF file format is not documented anywhere that I have been able to find on the web. But, looking at the files, they seem to be based on the TIFF image format, and they carry the TIFF file header.

Also, I have found that Dave Coffin has created an excellent utility for doing just what I want, dcraw. I could just as well just use that, but I think that it will be an interesting challenge to try to implement it myself. But I am quite sure that digging in to the dcraw source code will provide me with most of the details that I need to decode the format. 


Got a blog

So, I decided today that I needed a blog. So here it is, me blogging about programming in general, .NET and my everyday experiences. 

Why does the world need another blog; you might ask ? Well, it is an experiment for now. I will try to provide some interesting content - and at the same time it will be a place for me to document various findings and thoughts. Anyways, it is much better than the old "Web 0.1" static content website that I used to have on this domain ;-)

I encourage everyone to use the comments for questions and/or suggestions. 

Don't know who I am ? 

Well, you don't really need to, in order to read my posts. My name is Dennis Riis and I am a danish software developer / architect, currently working with web technologies. If you want to read more about me, you can always head over to the About me page.