Wednesday, September 17, 2014

Post Build ILMerge vs LibZ

For many years I used the ILMerge.exe utility in Visual Studio post build events to merge multiple .NET assemblies into a single assembly. This is most useful when you want to distribute an executable file and its dependent library files as a single EXE file. For Framework 4.0 projects I would add something this to the post build event:

"%ProgramFiles(x86)%\Microsoft\ILMerge\ILMerge.exe" /out:\\shared\utility\$(TargetFileName) /wildcards $(TargetFileName) *.dll /targetplatform:v4,%windir%\Microsoft.NET\Framework\v4.0.30319

I haven't tried using ILMerge on Framework 4.5 projects, but web searches hint that there are a few hurdles to getting it working.

It is well known that ILMerge does not work on WPF assemblies. The author says:

ILMerge is not able to merge WPF assemblies. They contain resources with encoded assembly identities. ILMerge is unable to deserialize the resources, modify the assembly identities, and then re-serialize them. Sorry!

In 2013 I stumbled upon an ILMerge replacement utility called LibZ (see Codeplex). The author explains the motivation for writing LibZ on the home page and has a nice technical discussion of how it works. Most importantly for me, LibZ has no problem with WPF assemblies. I have replaced all of my ILMerge post build commands like the one above with something like this:

xcopy $(TargetFileName) \\shared\utility /Y /D
libz inject-dll -a \\shared\utility\$(TargetFileName) -i *.dll


Notice that I copy the target file to the shared utility folder where it will finally live, then I process that file. I prefer to do that so the original build output file remains untouched.

Friday, September 12, 2014

Windows XP Updates Fail

I booted a Windows XP virtual machine for the first time in 6 months and found that it would not receive Windows Updates because the URL was not found. Web searches suggested strange and dangerous things such as deleting the ESE database of updates, registry hacks, system DLL renames, .NET Framework alterations and much more.

It turns out that you just have to visit this URL manually:

http://www.update.microsoft.com/microsoftupdate/v6/default.aspx?ln=en-us

I have no idea why such a monumental breaking change would occur so that the shortcut to Windows XP Update would send you to the wrong URL, especially without any obvious hint or redirection.

Tuesday, September 2, 2014

pkzipc extract to subfolder

I was trying to extract all files from a zip into a subfolder like this:

pkzipc -ext -dir \\shared\archive\140325.zip * tempfiles

But no matter what I did it kept extracting the files into the current folder. My command looked similar to the sample in the official PDF online documentation (except they used ".." as the output folder).

It turns out you have make the subfolder first, otherwise pkzipc just silently ignores the output folder name you specify and puts them in the current folder.

So the sample used ".." which always exists and therefore always works. The weirdly bad example wasted 15 minutes of my time because I thought I had the syntax subtly wrong.

03-Sep-2014

Sunday, August 31, 2014

Real Random Numbers

I was pleased to discover that there are many public web services available that provide real random numbers. You can easily generate pseudo random numbers in every modern programming language and platform, but these numbers are generated by deterministic algorithms and are not truly random.

Popular algorithms such as combined LCGs, the Mersenne Twister and the Subtractive Generator produce astronomically long sequences of pseudo random numbers that pass the toughest batteries of tests for randomness. So long as these algorithms are seeded and used cautiously they will satisfy most normal requirements. Be aware though that the internal state of these algorithms can be deduced by watching a certain number of sequential outputs, after which the sequence can be predicted forever. This predictability makes such pseudo random sequences unsuitable for use in cryptography.

When randomness is required in cryptography you should use a cryptographically secure pseudo-random number generator. Developers on the .NET platform can use the RNGCryptoServiceProvider class. Secure random numbers are slower to generate; my RandPlot application shows that a combined LCG can generate 2,300,000 numbers per second whereas the crypto secure class generates 165,000 per second. In practice this 14x speed difference probably won't be an issue because secure random numbers are usually used in small quantities for seeds or keys.

Thanks to online services we now have an exciting new alternative to pseudo or secure random numbers: real random numbers generated by natural processes. I found the following services attractive because they are backed by robust theory and they have friendly APIs to allow client applications to consume them.

ANU Quantum Random Numbers Server
The Australian National University generates random data in real-time at 5.7 GBits/sec by measuring quantum fluctuations of the vacuum. They provide a Web API that returns random data as JSON in three selectable formats.
random.org
Randomness is distilled out of atmospheric noise generated by radio receivers tuned between stations. There is an old Web API that returns data as text or XML and there is a new JSON API.
I've personally become quite attracted to the ANU's Quantum generator because of its futuristic flavour, the tantalising and trustworthy theory behind it, the blazing fast speed of the generator and the simple API.

The wonderful thing about using a service like the Quantum generator is that you never have to worry about even the tiniest flaws that may theoretically appear in random numbers generated by algorithms. You will never need to re-seed the generator. The previous century of detailed research and measurement of random number algorithms becomes a historical curiosity when you have access to real random numbers.

On the ANU's website they have link to a Sublight web page where you can find a C# class that mimics the standard .NET Random class, but it internally uses the Quantum web server as the source of randomness. I think the Sublight class is a bit clumsy and non-standard, so as a technical exercise I created the following two C# classes which I hope wrap and expose the Quantum service in a very natural and simple way.

QuantumStream.cs (download)
An unbuffered byte source derived from the standard Stream. You simply Read random bytes from the stream which is backed by the Quantum service.
Because the stream is unbuffered by design, you should wrap (decorate) the stream with a BufferedStream to improve performance when large amounts of random data are needed. See the Performance section below for an explanation.
using (var randbuff = new BufferedStream(new QuantumStream()))
{
  byte[] data = new byte[128];
  randbuff.Read(data, 0, data.Length);
  // use the real random data bytes
}

QuantumRandom.cs (download)
This class mimics the public members of the standard Random class. Some extra convenience methods are provided to return byte arrays, Int16, Int32 and Int64 values.
The random bytes are internally generated by a buffered stream that decorates the Quantum stream, as described in the previous section.
using (var qrand = new QuantumRandom())
{
  int n = qrand.Next();
  Console.WriteLine("{0,8:x8} {0}", n);
}

Performance Notes

Because the Quantum data is retrieved from a web service, there will be unpredictable delays in making the synchronous service calls. Tests show that there is a noticeable delay when making the first call with a new service client, but subsequent calls on that client have negligible delay.

If large amounts of random data are required, consider the following:
  • Keep an instance of a QuantumStream or QuantumRandom class alive for as long as necessary to avoid repeating the potential delay when making the first service call.
  • Construct a BufferedStream decorator or QuantumRandom class with buffer size larger than the default of 1024. The maximum buffer size is currently 128KB (131072 bytes). A larger buffer reduces the number of web service calls because more data can be retrieved by a single service call to fill the buffer on demand.
Greg (01-Sep-2014)



Setup project user logging

When working with Visual Studio Installer projects (vdproj project files) it's possible to create a class like this skeleton to perform custom actions during installer processing:

[RunInstaller(true)]
public class MyCustomActions : Installer
{
  public override void Install(IDictionary stateSaver)
  {
    // Custom install processing here
  }
  protected override void OnBeforeUninstall(IDictionary savedState)
  {
    // Custom uninstall processing here
  }
  // Other overrides are available
}

In the Setup project you open the Custom Actions Editor window and add the assembly containing the custom class to the event nodes. The methods will then be called at the appropriate points in installer processing.

It's possible to add custom dialogs to the install sequence, name the input fields and associate them with properties passed to the custom action events. This simple pre-packaged process is described in other web articles and is not the subject of this post.

An Installer derived class inherits the LogMessage method, but the output from this method does not seem to appear in the installer log. If you run msiexec.exe over an MSI file with the /L*v switch to produce the maximum possible verbose output, you will not see the LogMessage output.

For a long time I wondered where the LogMessage output went. Early last year I found an obscure hint in a web page (that I have lost) that says you can add something like this to the CustomActionData in the custom actions designer:

/logfile="C:\temp\custom-log.txt"

Friday, July 4, 2014

Dynamic C# code compilation

The ability to compile C# source code at runtime is a very useful technique that seems to be overlooked by a lot of .NET developers. It's a surprisingly easy way to add "scripting" abilities to your apps. I decided it was worth investigating while writing the HashStrings application which has to perform a dozen slightly different maths calculations and I wanted to be able to change them without the tedium of recompiling the project each time.

I have created a small example Visual Studio 2013 solution that shows how to dynamically compile a file of C# source code into an in-memory assembly, create and instance of a Type in that assembly and call one of its methods. Download a ZIP of the solution from:

http://www.orthogonal.com.au/computers/SampleCompile.zip (5KB)

An interesting trick in the project is to define the symbol ORDINARY which causes the project and source to compile in the simple traditional way. Without the symbol dynamic compilation takes place.

The important part of the example code is worth extracting and displaying here.

var provider = new CSharpCodeProvider();
var parameters = new CompilerParameters();
parameters.GenerateInMemory = true;
parameters.ReferencedAssemblies.Add("System.Core.dll");
parameters.GenerateExecutable = false;
parameters.CompilerOptions = "/define:ORDINARY";
string codeToCompile = File.ReadAllText(@"..\..\SourceCode\Worker.cs");
var results = provider.CompileAssemblyFromSource(parameters, codeToCompile);
if (results.Errors.HasErrors || results.Errors.HasWarnings)
{
  // Display the results.Error collection
  return;
}
Type t = results.CompiledAssembly.GetType("cscompile.Worker");
dynamic worker = Activator.CreateInstance(t);
worker.SayHello();

Don't forget though that there are other ways of generating dynamic code. See the MSDN articles titled Using Reflection Emit and Using the CodeDOM. These techniques are much more difficult but they are strongly-typed and more robust than simply feeding free form text into a compiler.

July 2014

AppDomains from libraries in subfolders

It's a nice deployment pattern to isolate "plug-in" code into library files in subfolders under the application folder and run these plug-ins in a separate AppDomain. By loading libraries and the assemblies they contain into a separate AppDomain it's possible to apply a restrictive security policy them and it's possible to unload them.

The Assembly Load, LoadFile and LoadFrom methods load libraries into one of the contexts in the AppDomain of the caller, and if this is the initial AppDomain of the Process then it cannot be unloaded until the Process terminates.

Organise your projects like this skeleton:

MyApp
MyApp.Plugin.Common
MyApp.Plugin.PluginOne
MyApp.Plugin.PluginTwo

The application and the plugin projects reference the common library, never each other. All of the plugins should implement an interface defined in the common library which defines their public contract. The plugin classes must be derived from MarshalByRefObject to allow strongly-typed communication via Remoting between the AppDomains.

When the application is deployed, arrange the folders like this:

Application Folder
  theapp.exe
  MyApp.Plugin.Common.dll
Application Folder\Subfolder One
  MyApp.Plugin.PluginOne.dll
  MyApp.Plugin.Common.dll
Application Folder\Subfolder Two
  MyApp.Plugin.PluginTwo.dll
  MyApp.Plugin.Common.dll

The application can search subfolders at runtime to find plugin libraries. You might use a folder naming convention or place a special XML file of instructions beside the plugins to identify and describe them. When a plugin is identified it can be loaded, called and unloaded like this:

var folder1 = new FileInfo("Subfolder One");
var ads1 = new AppDomainSetup();
ads1.ApplicationBase = folder1.FullName;
var dom1 = AppDomain.CreateDomain("Domain-1", null, ads1);
string file1 = Path.Combine(folder1.FullName, "MyApp.Plugin.PluginOne.dll");
string class1 = "MyApp.Plugin.PluginOne.ClassName1";
var plug1 = (IPlugin)dom1.CreateInstanceFromAndUnwrap(file1, class1);
Console.WriteLine(plug1.SayHello());
AppDomain.Unload(dom1);

This code creates an AppDomain with its base folder set to the subfolder where the plugin was found. It then uses CreateInstanceFromAndUnwrap passing the full path of the plugin's folder and the name of the class to instantiate in the AppDomain. The returned value (actually a proxy) is cast to the common interface and it can be called like a normal class method. The AppDomainSetup class has many other properties that help configure the AppDomain, such as specifying a config file.

The applications and the plugin projects do not reference each other and at runtime they only communicate using an interface over Remoting between the AppDomains.

While writing the real application that uses the technique described above I accidentally used the CreateInstanceAndUnwrap method (without the From) and I received misleading "Assembly not found" errors. It took me hours of suffering before I realised my dyslexic mistake caused by the similar names. The MSDN documentation describes the subtle differences between the two methods.

The technique described in this article is the manual way of implementing plugin isolation via AppDomains and is suitable for simple scenarios. The Managed Extensibility Framework (MEF) and the Managed Add-In Framework (MAF) are worth exploring for more complex scenarios.

July 2014