Precision Software Design, LLC

FileSystemWatcher Done Right.

2009-05-13T14:34:00.001-07:00

For some reason the FileSystemWatcher (aka ChangeNotify) in Windows is just not right. If you search for this in on the internet, you'll find tons of issues with people attempting to use it and running into problems. A couple of the top issues:

* The FileSystemWatcher calls multiple times on a single change.
* It does not call when certain programs change the file.
* If my process changes the file, the FileSystemWatcher calls me. How do I know I was the only one who changed it (and someone else didn't slip in after I called Close()).

I built the following class to encapsulate all of these issues, and to do file watching correctly.

The only things you need to know:
* Create a FileWatcher on a file on the disk. If it changes by an external process, you will get called once per change.
* If this process needs to write the file, call the FileWatcher's CloseFileInThisProcess() method when ready to close the file. This will ensure you don't get called when you change the file.

This is written for WPF. The only reason that matters is because of the threading issues. It can easily be tweaked for WinForms.

There is an example usage following the code.


    /// 
    /// This filewatcher works around the 'issues'
    /// with Window's file watcher .NET FileSystemWatcher
    /// or Win32::ChangeNotify).  This code is written for
    /// WPF, but can be tweaked to work with Winforms as well.
    /// 
    /// It solves these two main problems:
    /// * The FileSystemWatcher calls multiple times on
    ///   a single change.
    /// * If my process changes the file, the
    ///   FileSystemWatcher calls me.
    /// 
    /// It solves the former by using a 100 ms timer to
    /// collapse multiple calls into a single call.  It
    /// solves the latter by storing the file size and
    /// time when this process writes a file, and
    /// comparing this to the values when notified by
    /// FileSystemWatcher.
    /// 
    /// Usage is straightforward, except that you must
    /// call CloseFileInThisProcess when you are closing
    /// the file that this watcher is watching.  It will
    /// carefully close the file in such a way that it
    /// can later tell if the change was by this process
    /// or another.
    /// 
    /// 

    public class FileWatcher : FileSystemWatcher
    {
        public delegate void FileChangedHandler(string filepath);
        public FileWatcher(string filepath, FileChangedHandler handler) : 
            base(System.IO.Path.GetDirectoryName(filepath), 
            System.IO.Path.GetFileName(filepath))
        {
            FilePath = filepath;
            Handler = handler;
            NotifyFilter =
                NotifyFilters.FileName |
                NotifyFilters.Attributes |
                NotifyFilters.LastAccess |
                NotifyFilters.LastWrite |
                NotifyFilters.Security |
                NotifyFilters.Size;
            Changed += new FileSystemEventHandler(delegate(object sender, FileSystemEventArgs e)
            {
                System.Windows.Application.Current.Dispatcher.BeginInvoke(
                    new VoidDelegate(this.FileChanged));
            });
            UpdateFileInfo();
            Timer = new Timer(100);
            Timer.AutoReset = false;
            Timer.Elapsed += new ElapsedEventHandler(delegate(object sender, ElapsedEventArgs e)
            {
                System.Windows.Application.Current.Dispatcher.BeginInvoke(
                    new VoidDelegate(this.TimerElapsed));
            });
            EnableRaisingEvents = true;
        }

        /// 
        /// This only works with StreamWriters.  You may need
        /// to make your own version for whatever writer you are
        /// using.
        /// 
        /// How this works:  It stores the file size before
        /// calling close.  Then, after close it grabs the write
        /// time.  There is a minute corner case here:  If
        /// another process gets in and writes the file after
        /// the close, but before the GetLastWriteTime call, and the 
        /// file is the same length, we will not detect the change.  
        /// If that is critical, one could do a checksum on the file....
        /// 

        public void CloseFileInThisProcess(StreamWriter writer)
        {
            writer.Flush();
            LastFileLength = writer.BaseStream.Length;
            writer.Close();
            LastWriteTime = File.GetLastWriteTime(FilePath);
        }


        void UpdateFileInfo()
        {
            var fileInfo = new FileInfo(FilePath);
            LastWriteTime = fileInfo.LastWriteTime;
            LastFileLength = fileInfo.Length;
        }

        public delegate void VoidDelegate();        
                
        void FileChanged()
        {
            var fileInfo = new FileInfo(FilePath);
            if (LastWriteTime != fileInfo.LastWriteTime || LastFileLength != fileInfo.Length)
                if (!Timer.Enabled)
                    Timer.Start();            
        }

        void TimerElapsed()
        {
            UpdateFileInfo();
            Handler(FilePath);
        }
        
        string FilePath;
        FileChangedHandler Handler;
        DateTime LastWriteTime;
        long LastFileLength;
        Timer Timer;
    }

Here is an example usage:


    public partial class Window1 : Window
    {
        public Window1()
        {
            InitializeComponent();
            
            // Set up the file watcher given a filename and a handler.
            Watcher = new FileWatcher(@"c:\temp\test.txt", FileChanged);            
        }
        FileWatcher Watcher;
        void FileChanged(string filename)
        {
            // For now just display a message when the file changed.
            MessageBox.Show("Got here");
        }        
        
        // Here is an example of writing the file.
        protected override void  OnMouseDown(MouseButtonEventArgs e)
        {
            var writer = new StreamWriter(@"c:\temp\test.txt");
            try
            {
                writer.Write("Hello, world, number " + I++);
            }
            finally
            {
                Watcher.CloseFileInThisProcess(writer);
            }
        }
        int I = 0;        
    }

Natural Language .NET Date Parser.

2009-03-09T14:23:00.000-07:00

As part of the Smpl application, we (Precision Software) built a natural language date and time parser for .NET. This is not just another DateTime.Parse("...") class. It is built to handle the majority of ways that humans communicate dates, times, date spans, time spans, etc. in English. And, by the way, English is a much simpler and compact way to communicate complex time specifications than form based UI. If you don't believe me, take a look at Outlook's Recurring date form. Wow. All this to describe what English describes in a few words (ex: "The 9th of every month").

Have you needed a date parser that can handle recurring (repeating) dates?
How about one that can parse something like, "Every Thursday, 2 PM".
Need to parse dates strings that occur once per month, like "The first Tuesday of every month". If you searched the web, you probably found the Perl Date::Manip module. If you are writing a Unix script, that is great, but for those of us C#, Visual Basic, or other .NET languages, calling Perl to parse dates is not a practical option.

This parser treats all dates and times as spans, with optional recurrence. Thus, "Tuesday, 10AM" is treated as a span from 10-11AM. This creates a really convenient API: date.Overlaps(otherDate) will check if two dates overlap. This will return true, for example, if date = "Jan 5", and otherDate = "Tuesday, 10AM", assuming this Tuesday is January 5th.

The optional recurrence allows enter dates in the form of "Every Monday, 8PM", or "Every January 5", "Every January", or even "Everyday, 5-9PM". To iterate these recurrences, one simply asks for date.Next("today"), for example, which will return the next occurrences of a given date after today. Combining this with Overlaps makes a very tight, easy to read program.

To deal with ambiguous situations, the parser has a set of flags to control preferences. For example, "Tuesday from 3-5 PM" could mean *this* Tuesday, or it could mean *every* Tuesday. By default, this causes a parse error, but by preferring recurring or not, the error can be eliminated. In such a case, one can still, for example, explicitly state "Every Tuesday, 3-5PM".

The smart date parser is a .NET module, and therefore can be used with any .NET language, including ASP and other web based languages.

You can see it in action here.

If this module would be of use to you, let us know by e-mail: contact@precisionsoftware.us. We will respond within 1-2 business days of your e-mail. For faster response, call us at 208-882-5980.

Specification Authoring Tool

2009-01-13T14:39:00.001-08:00

Does anyone else out there find it annoying that there is not a good specification authoring application on the market? I'm looking for an application that allows one to author requirements specifications for anything from software to airplanes.

One would argue that Microsoft Word is the universal document authoring tool. Why would you want a different tool? For those that do development primarily on computers (software developers, electrical engineers, etc.), Word doesn't work very well. Some of its drawbacks:
* It has a binary file format. This doesn't play well with version control. The new XML file format is better, but its not text. It won't merge. Their diff support is also poor.
* Its styles are horrible. If you try to use them the way they are intended to be used, you get very frustrated.
* It has no document consistency checking. Its all Greek to the computer.
* It doesn't help you author specifications using your format.
* Anyone that has used Word with a file locking version control system knows that Word is broke. Word nicely doesn't inform you that the file you are editing is locked until AFTER you've made changes--ARGH!!!
* And finally, you cannot programmatically get data out of a word document!! The most important information about your product is stored in a format that allows you to do two things: Read it (including print, e-mail, etc), and Text search it (thanks to Google, who had to come along and make it practical). Why can't I generate manuals by pulling out features from specifications? Why can't I generate or configure parts of the product directly from the specifications?

Is it too much to ask for a tool that meets the above? The tool should also support:
* Outlines
* Tables
* Styles--done right
* Embedded images
* Ability to publish directly to HTML/PDF

All while storing the source in a plain text, human readable (not geek readable, like XML) file. I think most of us would drop Word in a heartbeat if such a tool came along.

Smpl Dogfooding.

2008-11-17T15:16:00.000-08:00

dogfooding (verb) - In a nutshell, dogfooding means "using your own product". A product which is being dogfooded tends to be a lot more polished. When a normal user is annoyed by the product, they can't do anything about it. But when a developer is annoyed by the product, they can stop what they are doing and make the product less annoying. (From http://www.urbandictionary.com)

What a month it has been at Precision Software!

Since the completion of milestone 1, we've been using Smpl for a variety of tasks. We now have 20+ real world use cases demonstrating Smpl for uses such as:
* Test plans and test generation
* C# code generation (Menus)
* Church organization (members, sunday school classes, ministries, mailing lists, reminders, directory, and web site)
* Object relational mapping (ORM)
* Generating word documents
* Sending e-mails
* Web site generation
* VHDL module entity generation
* State machine VHDL generation

Check out the updated web site. By the way, it is now being generated by Smpl. I sat down to update the site, and decided I was not going to stare at PHP any longer! So, I copied the content into Smpl, and in less than an hour had it regenerating the site. Of course, getting all the details took a bit more time. However, I can now edit something at a high level, instead of dealing with PHP directly. The Smpl translation templates handle creating the menus, inserting tables, bullets, etc.

What I love about Smpl is you only need to abstract what you get benefit from. In the middle of the Smpl web site, I put HTML tags. Smpl doesn't care--just passes them through. However, I have it generate the tables and bullets on each page so I don't have to think about that.

The rest of this post is the Precision Software web site (http://www.precisionsoftwaredesign.com) content, in Smpl. I'm sure it doesn't word-wrap very well for this post (and it looks like some of the HTML tags are getting interpreted), but at least you get the idea.

Precision Software Site :Web Site
Page:
Mission
Description:
Eliminating the tedious work that people do on computers.
Above Bullets:
How much of your time do you spend...
Bullet:
Engineering and Development
Description:
...making 20+ changes to specifications, designs, code, and tests due to a single requirement change.
...causing bugs because you don't know every place that needs to be changed.
...dealing with common mistakes made by new members of the team because there was no way to enforce agreed upon rules.
...making the same change to the 10 products in a product family.
Marketing and Sales
Description:
...maintaining four version of a PowerPoint presentation.
...updating the data sheet, product brief, product manual, and web site when a product feature changes.
...trying to fit your data into the rows and columns of an Excel spreadsheet.
Business
Description:
...developing a $100K+ custom database application, and by the time the application is ready your process has changed and the application must be updated.
IT
Description:
...automating everything that you can--but knowing there are more major tasks that you know the computer could be doing

Products
Description:
The Smpl compiler is focused on technical information, while the Smpl Information Processor is the future of Smpl, making it available for non-technical information as well.

Bullet:
`Smpl` Compiler
Bullet Link: Smpl_for_DSM.pdf
Description:
The Smpl compiler reads Smpl model definitions, models, and translation templates, and then processes and executes the desired result--anything from generating C# code to sending an e-mail with a Word document attachment.
The Smpl compiler enables developers to implement Domain Specific Modeling in their development processes. Steven Kelly wrote an excellent book on this topic, and see also the DSM Forum web-site.
`Smpl` will move DSM into the mainstream. This whitepaper discusses how: Smpl for DSM. This paper shows examples of what can be done with Smpl: Smpl Examples.
The compiler is currently in pre-Alpha testing. Please contact Precision Software at 208-882-5980 if you are interested.

Smpl Information Processor
Bullet Link: Smpl_Information_Processor.pdf
Description:
Let's face it, today's word processors are dumb. They have turned computers executing 3 billion instructions per second into advanced typewriters. Spreadsheets are a step up, but very little real world data actually fits into its rows and columns. Finally, databases are anything but user friendly.
Smpl is the future of information processing. This whitepaper discusses that future: Smpl Information Processor. This product is not yet available.

Dsharper
Bullet Link: dsharper.php
Description:
In the process of developing the Smpl compiler, this C# to D programming language convertor was developed. It is released open source to the C# and D communities. Click here for more information.

Technology
Bullet:
Smpl :TM
Description:
Imagine if every document you authored in the computer was understood by the computer--every document was effectively a mini-database. If the computer can understand your documents, it can do the mundane work you don't like doing. This leaves you to do the tasks that you enjoy.
In technical terms, the Simple Modeling and Processing Language (`Smpl`) is a textual domain specific modeling language and template based compiler that enables computer aided authoring typical of source code editors in IDEs but generalized for non-programs, and also brings the sought after goals of code generation and automation within reach.
Smpl is a common human/computer language. Instead of the plethora of computer languages (including the explosion of such languages that domain specific languages enables), Smpl is one language. Like natural language, Smpl's extensibility comes through defining new terms, not changing the rules of the language. A diagram may be worth a thousand words, but it also takes a thousand clicks to create and update. Using text (or words) is, and will be, the most efficient way to communicate with a computer.
The following white paper introduces the Smpl technology: Introduction to Smpl. This paper shows examples of what can be done with Smpl: Smpl Examples. Smpl technology is patent pending.
Advantages
Description:
More efficient authoring of specifications, designs, code, tests, web and published document content, etc.
Information consistency and correctness. The computer is finding and helping fix mistakes as you type.
Queriability. The query power of databases available on your documents.
Automation. Sending e-mails, updating web site content, notifications of information changes.
Generation. Generate documents, source code and scripts, directories, zip files, etc.

(Blog)

About Us
Description:
Precision Software was founded in 2004 as a research company investigating methods of improving software development. The founders are listed to the right.

Bullet:
Eric Hewitt
Description:
Eric graduated Summa Cum Laude from the University of Idaho with a degree in Electrical Engineering. He worked with Advanced Hardware Architectures (AHA), an Integrated Circuit design firm, for seven years, where he gained experience with all phases of IC development including specification, design, testing, and project management. Eric then worked with Schweitzer Engineering Laboratories (SEL), writing embedded software for their power system protective relays before joining Precision Software Design.
Image: eric.small.jpg
Michael Hewitt
Description:
Michael graduated from Rice University in 1993 with a BA in Computer Science. For 10 years, he worked as a programmer and lead programmer at Microsoft on projects including Windows 95, the first four versions of Outlook, and a ground-breaking development environment in Microsoft Research called Intentional Programming. He started with Precision Software Design in 2004.
Image: michael.small.jpg

Contact
Description:
If you have questions about Precision Software technology or products, please send e-mail to: contact@precisionsoftware.us

Bullet:
Address and Phone
Description:
Precision Software Design, L.L.C.
105 E. 2nd St. Suite 1
Moscow, ID 83843
Phone: 208-882-5980

Smpl Compiler Milestone 1 Complete!

2008-10-08T15:34:00.000-07:00

This blog post is a bit late. Our plan for completion of milestone 1 on the compiler was the end of August, 2008. We finished a couple of weeks late, about mid September. Not bad, considering that at one point our schedule showed completion in 2009!

Milestone 1 included the highest risk areas:

Nearly all of the Smpl lexer/parser, including basic table support.
The start of a type checker.
The basic engine features necessary to do something with Smpl: Schema, category membership, templates, basic functions.
Textual output.

How does it look? Good. Real good. No, I mean, really, really good!

We're now building demos. Here's some examples of how good:

DSL/DSM/Code Generation:

Hardware module specification. This demonstrates using Smpl to model an ASIC VHDL or Verilog module, including description, dates, Generics, and Interfaces (Ports). The Smpl schema for this example was 18 lines of code. The generation of a VHDL module entity from a Smpl model is 40 lines of code. The generation of a professional Tex document specification from the model used 30 lines of Smpl code. In none of this code did I have to think about how to lex or parse my DSL, or how to construct the abstract syntax tree, or how to patch up references, or any of the stuff you usually have to do when writing a DSL. This entire process would take about 1-2 hours!
Object Relational Mapping. We created an object oriented Smpl model of a database. From this model, we do a two step translation (ORM Model to SQL Model, then SQL Model to SQL Text) that generates the SQL code that creates the database. ORM Schema: 20 lines; SQL Schema: 20 lines; Translation from ORM to SQL: 50 lines; Translation from SQL Model to SQL Text: 45 lines. This is just the beginning (it doesn't handle queries yet), but it creates all tables and foreign keys to handle object inheritance and references. That's only 135 lines of code, ritten in a few hours!

How about small business/organization automation:

A consulting business' client list, hour logs, and billing. This application uses Smpl to model client's contact information, projects, contract details, and consultant timesheets. Foreach project, it will generate an e-mail to each client containing a Word document attachment showing hours, rate, and total bill. Yes, a Microsoft Word Document (thanks for .docx, Microsoft!) Schema: 18 lines: translation to E-mail & Word Document: 70 lines. That includes the text of the e-mail message! Oh yeah, and I should mention that the word document template (Smplified) is 78 lines of code--but I didn't have to write that--smplc generated it from the .docx file for me!
Church organization model. This allows describing the parts of the organization, the members and what parts they are involved in, and a calendar, and generates: E-mail lists for all, parts, or specific members (either by name or by title) of the organization, Reminder e-mails, Web site, and a Word Document directory. The entire Smpl code is roughly 250 lines. Wow!

What does the Smpl look like, you ask? I'm not yet ready to present the details, but I will give you a couple of example models. The following is real code. Read by the computer. Processed and generating the above described artifacts.

An up/down counter module specification in Smpl looks like:


Up Down Count : Module
  Description:
    This module is a simple n bit unsigned counter, where n is a generic,
    supports counting up, down, or pausing.  The counter can be reset, count
    up, or count down based on inputs.
  Initial Version: 5/21/2008
  Generic:
    Bits
      Description: The number of bits used to represent the count value.

    Port:              Direction  Width    Description
    --------------------------------------------------------------------------------
      Clock            in                  System clock.
      Reset            in                  Reset the count.
      Count Up         in                  If true, the counter counts up each clock cycle.
      Count Down       in                  If true (and Count Up is false), counts down.
      Count            out        Bits     Current count value.

A consultant's client and billing data looks like:


Company:
  Boeing
    Address:  1 Boeing Lane
    City State Zip: Tacoma, WA  98765

Project:
  Superconductor Research
    Contact: Bill Jones
    Contact Email: bill@yo.com
    Client: Boeing
    Under Contract:
      Consultant      Rate
      --------------------
      Phil Smith      95
      Bob Wilson      110

  Engine Mount Design
    Contact: Tom Smith
    Contact Email: tom@dude.com
    Client: Boeing
    Under Contract:
      Consultant      Rate
      --------------------
      Bob Wilson      90

:Hour Log
 For: Phil Smith
 Entry:
   Date   For Project              Hours
   -------------------------------------
   9/2    Superconductor Research  4
   9/12   Superconductor Research  8
   9/24   Superconductor Research  3

:Hour Log
 For: Bob Wilson
 Entry:
   Date   For Project              Hours
   -------------------------------------
   9/3    Superconductor Research  3
   9/12   Engine Mount Design      9
   9/21   Superconductor Research  4

There is still a lot of work ahead. We're looking for professionals to join us, especially on the marketing side. Its time to take this technology out of the lab, and make it available to the masses.

The difference between a specification and an implementation

2008-07-10T09:02:00.000-07:00

In some ideal world, the two would be the same--you would write a specification, and the computer would build it! However, if that were true, then it would be impossible to create something new. Why? Because if you can’t describe what something is, you can’t build it. Any system that allows the description of a specification with automatic implementation must have a fixed set of terms.

The first time something is done, the step from specification to implementation requires faith. “Let there be light”. What is light? Light has never existed before. So, how can it now exist? Let there be light is a fine specification. But, the implementation cannot be directly derived from the specification. Something has to be added.

Generally, specifications are easy to understand, hard to believe. It’s easy to understand the statement ‘Build a computer that runs at 10 TeraHz’, or ‘Build a spaceship that will fly to the nearest solar system outside of our own.’ It’s hard to believe either can be done.

Implementations are hard to understand, easy to believe—because you’re looking at the thing in action! It’s easy to believe there is light. We are living in the implementation of that specification. Figuring out exactly how light works, on the other hand, that’s hard to understand.

Another example was Bill Gates and Paul Allen. They had a specification to build a BASIC interpreter on the Altair 8800. The specification was simple: Those words, and the BASIC language specification. Easy to understand, hard to believe. Once they completed the implementation, it was easy to believe they had it working.

Now, if you have done something before, then yes, it is possible to join the specification and implementation. You are not really doing anything new, however. You are just doing the same thing over in a modified way. However, if you want to add anything new to the product, specification is again required.

Of course, you can always skip the specification stage, and just write the implementation. This is inefficient. Because you don’t fully understand what you are building, you will undoubtedly make mistakes. Finding mistakes at the specification stage is much cheaper than at the coding staging. Finding mistakes at the coding stage is much cheaper than at the testing stage. Finally, finding mistakes at the testing stage is much cheaper than finding them in the hands of the customer.

Interestingly enough, it should be possible to define a language, and be using it, without thought for what exactly the terms in the language will do. Defining a language should take seconds, not hours, days, or weeks! Creating an implementation (or describing how to create an implementation from a specification), of course, takes time. Again, it’s easy to ‘understand’ a specification. It’s difficult to implement it.

What is needed, then, is a system that allows you to formalize and implement what you have done before, while simultaneously mixing this with specification for what is new in a particular product. Your specification authors, when describing how to display information or what controls exist on a product, will use the formalized specification/implementation language. When describing how the new fangled space travel feature will work, they will use more free-form. The computer consumes the formal parts, and produces code. You, the developer, get to formalize the informal parts, and describe their implementation to the computer. Next time someone needs the space travel feature, they now have a formal language/implementation. Our hope at Precision Software is that we are building exactly that in the language we call Smpl.

I'll close this post with these words: Without faith it is impossible to please God!

Smpl Development Has Begun!

2008-05-09T10:19:00.001-07:00

Well, after nearly 4 years of (both technology and market) research, we are finally starting development on the product we set out to build so many years ago.

Our grand ideas of storing all product development information in a single database, providing a word processor-like interface to this database, allowing abstraction across all stored information, and transforming the information into code or other artifacts have been set aside.

The real world has humbled us. However, it has also brought out the best in us. We have been squeezed to create exactly and only what is necessary. We're calling it a Simple Modeling and Processing Language, or Smpl for short. It is a textual domain specific modeling language and associated translation based compiler.

It will be thin, light, simple, and yet powerful. Imagine Microsoft DSL Tools or MetaCase MetaEdit without all of the diagram and GUI overhead. It is text--so, yes, it will integrate with all of your existing development tools and processes. You will be able to use it in preprocessor mode, allowing you to smoothly transition from a traditional manual coding approach to full domain specific development.

We are confident you'll like what you see. The new web site will be coming along shortly, along with a two page introduction to Smpl technology. Development is moving forward. Alpha release is expected late 2008.

When You Are Researching Vs. Developing

2008-04-10T17:32:00.000-07:00

A completed (thorough and consistent) specification marks the end of research, and the beginning of development.

Prior to a completed specification, one is doing research (no matter how short or long this time may be). There are still unknowns. Unknowns require research. Research time is not predictable.

After a completed specification, one is doing development. The unknowns are hopefully minimal. Research is not required. Development time is predictable. An incomplete specification will allow research to slip into the development phase, thus making development time unpredictable.

Literate Programming

2008-02-20T12:09:00.000-08:00

The primary goal of literate programming is to create well documented code. Accordingly, a literate programmer simultaneously authors both the documentation and the code. Macros enable both the abstraction and a more intuitive ordering of code. Abstraction is performed primarily to improve understanding, secondarily to foster reuse. Compilation of the documentation (called “weaving”) produces a well formatted TeX document which contains the embedded code as well, ordered in an intuitive fashion. Compilation of the code (called “tangling”) is a single-level process whereby documentation is eliminated, macros are expanded, and the desired code is thereby generated. Macros have similar capabilities and drawbacks of C macros. CWEB, in fact, employs C macros to do much of its work. The primary advantage of literate programming is nicely formatted, readily approachable documentation of the code base.

Literate programming is missing the following components of a Template Oriented Programming (TOP) system:

Model syntax
Multi-level model to model translation
Custom errors to identify model misuse
Model queries
Translation debugging

Most of the CWEB manual focuses on formatting of documentation and code, whereas most of a TOP system focuses on model translation. Only the lowest level textual output of a TOP translation focuses on formatting. In a TOP system, the model serves as the documentation. A model pretty-printer can create a well-formatted, hyperlinked, read-only view of the model. Syntax highlighting within a text editor can be employed to create a reasonably formatted writable version of the model.

Literate programming can be a useful way to organize code. It is hoped that a TOP system will provide many of the benefits of literate programming, but also yield a considerably higher return on investment with respect to abstraction. For example, a literate programmer must always be aware of the underlying code being produced. In contrast, a TOP programmer should be able to think at the level of the model being created, for the most part ignoring the translation of that model to lower level model/code.

Template Oriented Programming

2008-02-18T22:55:00.000-08:00

The phrase Template Oriented Programming is new. If you don’t believe me, type that quoted phrase into Google and you’ll find only about 5 unique hits! What is template oriented programming, you ask. This blog post seeks to answer that, and will hopefully, in part, make its way to Wikipedia, and, with any luck at all, to CS201 at your favorite educational institution.

Template oriented programming (TOP), as its title hints, is a programming paradigm that focuses on templates to accomplish a programmer’s goals. Note that when I use the word template here, I am not talking about a specific programming specific language feature such as (C++ template, C# generics, etc). The goal of template oriented programming is not to directly create machine code (or byte codes) to be executed on a (virtual) machine. Instead, the output of TOP is information to be processed by other code (or by humans). This could include familiar template output such as HTML/XML, or more complex code like C/C++/Java/C#, shell scripts, VHDL/Verilog, CAD scripts, etc. TOP allows one to generate one or more tedious artifacts from a simple, user defined model that is input.

Similar to the paradigm shift that occurred when moving from structured programming to object oriented programming, or from unmanaged to managed, garbage collected code, programming in the TOP paradigm requires a different way of looking at the information. The programmer starts with what he desires to have as output of a given set of code. This output is then abstracted as necessary to allow customization based on the input. The abstractions include: Simple field substitution similar to word processing form letters, extracting parts of the template into sub-templates that can be reused, looping over part of a template, making part of a template conditional, etc. The paradigm shift requires thinking about what should be produced instead of thinking about what the computer should do step by step.

This discussion leads to a short rabbit trail. Imperative programming languages (Fortran, BASIC, and C for example), evolved from machine language, and focus on telling the computer what to do. Declarative programming languages (Lisp, Prolog, XML), on the other hand, describe relationships between information. The drawback of pure declarative languages is that they cannot do anything, since they have no imperative aspect. Thus, most declarative languages have some way to start or otherwise control the execution of the program.

TOP applies the technique of declarative programming to code generation. This allows things like referring to information that does “not yet” exist. Such references, when first evaluated, will be null. However, when the information is “later” created, the reference will be created correctly. Notice the time related words were placed in quotes. That is because, in TOP one does not think about time. The processing order is handled by the system. Relieving this burden from the programmer eliminates many common bugs, and frees the mind of the programmer to focus on the more valuable, customer driven requirements.

A simple example is in order at this point. Let’s say a program was to evaluate a parenthesized mathematical expression. Assume the expression is mapped to a tree, with the leaves being the inner most expressions. A traditional imperative evaluator must find the leaves, evaluate these first, and then work up. A declarative solution, on the other hand, can simply state the desired result for each operator. All sub expressions are evaluated independently, and parent expression results will eventually be updated whenever a sub expression result changes. The system is responsible for making sure the result at the top is correct based on the last changes to any sub-expressions.

Declarative programming solutions have predominantly been used only in academia. They have many problems that make them inadequate for most commercial purposes. These issues, however, are mostly related to the runtime use of a declarative system: Difficulty handling the updating of program state, performance issues associated with information updating, etc. The TOP paradigm, instead, uses declarative techniques at compile time, avoiding many of these drawbacks.

This field is similar to that of template meta-programming. Template meta-programming systems are generally applied to a single, existing language. C++ templates, for example, allow adding templates to the C++ language. The focus is how to add compile time abstraction to the C++ language. TOP, on the other hand focuses on the templates, with the output language (or languages) being a secondary concern. In addition, the target language is defined by the programmer.

Template metaprogramming and template processing are not new. What is Precision Software doing that will breath life into these currently niche paradigms? Template solutions today focus on the getting the characters in the right order to produce a computer program. Our solution, however, focuses on a model of the information that is not character based. In this respect, it is similar to Lisp. The system supports translating between input and output, where neither are text—both are model. The closest analogy is to imagine operating on the abstract syntax tree within a compiler. This frees the programmer from having to continually parse the input, and unparsed (convert to text) the output. This approach allows information to be represented in a normalized fashion, translated with one or more stages within the tree format very concisely, and then output to one or more target artifacts.

Intentional Programming

2008-02-08T14:57:00.000-08:00

On the surface, our work would seem very similar to Intentional Programming (IP). However, there are some important differences. IP allows for the creation of arbitrary constructs with unconstrained semantics and unconstrained visualization. At the time I worked in IP, both reduction (R3) and visualization were constructed with imperative code. Having this level of power definitely falls in the category of creating radically new “languages”, in the same sense that imperative textual compilers are capable of create new languages (e.g. C++, C#, D, Java, Smalltalk, etc).

What we provide is a shared fabric for abstraction. Our semantics and visualization are constrained. All information is represented using a common model, and that model is visualized uniformly with only occasional deviations for brevity. Once information is represented as model, templates may be used to abstract that model in a controlled, type-checked fashion. Templates are used both to abstract above a model and to translate higher level model semantics into lower level model semantics.

Due to constrained visualization, our system will not work well as a structured GPL editor. This is one of IP's goals, but is not one of our goals. Instead, we believe that GPL code is stored and edited quite well as text. Whereas an IP code base may consist of quite a lot of structured GPL (this is, in fact, exactly what the IP code base itself does consist of), a code base in our system will consist of layers with a handful of high-level abstractions at the top and textual GPL code at the lowest level leaves. Terse visualization of model is therefore less important, and a uniform way to view and abstract every level is far more important.

One lesson we've learned is that abstraction breaks fancy visualization. Since our whole world is about abstraction, visualization must be quite general and not depend on a fully realized model structure because the model structure within any given template may only be partially realized. On the other hand, the editor is completely context aware, so that even within a highly abstracted model, it can still intelligently prompt for valid relations and termini. Also, templates at all levels must conform to the syntax within which they are operating, yielding a "type-checked" expansion ("reduction" in IP terminology) from top to bottom.