Regular Expressions are a great tool for programmers on any platform. We aim to highlight the skills that work well for programmers on any system and this is definitely one of them. Regular Expressions are not black magic, they are very helpful and when it comes to templating, highlighting or any kind of searching textual content they are the best tool for the job.

Some Platforms or Systems that use Regex

• The .NET Framework
• ActionScript
• ColdFusion
• Java
• JavaScript
• Perl
• PHP
• Python
• Ruby
• Tcl
• Apache HTTP Server
• Google Analytics
• Google Code Search
• Microsoft Exchange Server
• Windows PowerShell
• OpenOffice.org
• Oracle Database
• Any decent programmer’s text editor or IDE
• And thousands more…

One of the best regular expression sites is just http://www.regular-expressions.info/ but there are many. If you are new to regular expressions or finally want to start understanding what makes them work

Some samples of common programming tasks:

Example of White Space Trimming

You can easily trim unnecessary whitespace from the start and the end of a string or the lines in a text file by doing a regex search-and-replace. Search for ^[ \t]+ and replace with nothing to delete leading whitespace (spaces and tabs). Search for [ \t]+$ to trim trailing whitespace. Do both by combining the regular expressions into ^[ \t]+|[ \t]+$ . Instead of [ \t] which matches a space or a tab, you can expand the character class into [ \t\r\n] if you also want to strip line breaks. Or you can use the shorthand \s instead.

Grabbing HTML tags or Markup

<TAG\b[^>]*>(.*?)</TAG> matches the opening and closing pair of a specific HTML tag. Anything between the tags is captured into the first backreference. The question mark in the regex makes the star lazy, to make sure it stops before the first closing tag rather than before the last, like a greedy star would do. This regex will not properly match tags nested inside themselves, like in <TAG>one<TAG>two</TAG>one</TAG>.

<([A-Z][A-Z0-9]*)\b[^>]*>(.*?)</\1> will match the opening and closing pair of any HTML tag. Be sure to turn off case sensitivity. The key in this solution is the use of the backreference \1 in the regex. Anything between the tags is captured into the second backreference. This solution will also not match tags nested in themselves.

This is just a few great uses. When you build your regex libraries, they never die, they span all platforms.

Check out ten great reasons to learn and use regular expressions. Also the javascript RegexPal.

Pseudo-code for using regex in a recursive way

When used by themselves, these regular expressions may not have the intended result. If a comment appears inside a string, the comment regex will consider the text inside the string as a comment. The string regex will also match strings inside comments. The solution is to use more than one regular expression, like in this pseudo-code:

GlobalStartPosition := 0;
while GlobalStartPosition < LengthOfText do
  GlobalMatchPosition := LengthOfText;
  MatchedRegEx := NULL;
  foreach RegEx in RegExList do
    RegEx.StartPosition := GlobalStartPosition;
    if RegEx.Match and RegEx.MatchPosition < GlobalMatchPosition then
      MatchedRegEx := RegEx;
      GlobalMatchPosition := RegEx.MatchPosition;
    endif
  endforeach
  if MatchedRegEx <> NULL then
    // At this point, MatchedRegEx indicates which regex matched
    // and you can do whatever processing you want depending on
    // which regex actually matched.
  endif
  GlobalStartPosition := GlobalMatchPosition;
endwhile

Regex makes you a better solution provider, it give the engineer the tools to transcend platforms with ease. They help to create a baseplane solution in technology and a market platform.

Sun has been on a rampage lately. They recently purchased MySQL (which has some questions with InnoDB) but they also are supporting Python and integrating it into the VM offerings with Jython.

So now we have Google (They employ Guido), Microsoft (IronPython) and Sun (Jython) all turning into Python-istas. Does this mean Python is destined for greatness in the near future? Well I recommend learning it. It is a great language and it has a high productivity rate. Time will tell if it has the ability to be in extremely large code bases. I think it is just a matter of the architecture and organization as with any project.

Jonathan Schwartz at Sun mentioned also taking the “J” out of “JVM” to just make a VM much like a .NET framework. So Microsoft copied Java with C# the JVM and added multiple languages. Then Sun comes back and added multiple byte code compiled languages. And then they both focus on dynamic engines to implement versions of Python, Microsoft doing this within their DLR.

Many times the larger market languages that end up running business or “enterprisey” are heavily influenced by companies, in addition to the other demands from consumers, which here is programmers. So when the big three are all banking on Python as a draw there is a definite market draw there and a tell on the future. But I think it is apparent that Python is becoming a standard market baseplane language.

It is great to be able to use other libraries from .net with IronPython and java libraries with Jython. It has found a way to integrate with the current infrastructure and the language has low bar entry but deep benefits. Python is snaking its way into the market.

Python is so non verbose…

import sys
import clr
from System.IO import Path, Directory, FileInfodir = Path.Combine(sys.prefix, 'DLLs')

if Directory.Exists(dir):
    sys.path.append(dir)
    files = Directory.GetFiles(dir)
    for file in files:
        if file.lower().endswith('.dll'):
            try:
                clr.AddReference(FileInfo(file).Name)
            except:
                pass

This sample is from IronPython showing adding dynamic references to all dlls loaded.

Check out this HTML/XHTML parser in pure Python.

from HTMLParser import HTMLParserclass MyHTMLParser(HTMLParser):
def handle_starttag(self, tag, attrs):
print "Encountered the beginning of a %s tag" % tag
def handle_endtag(self, tag):
print "Encountered the end of a %s tag" % tag


XML is railed against plenty these days for being too verbose and leads to massive config invasions in your codez. But it deserves due respect for what it did.

What did it do you say?

Well, there was a time when data services rarely existed, connecting to trading partners or business partners was almost impossible. Connecting with partners directly to their RDBMS which is poor coupling and is not as message based as services.

Before XML was accepted it was a pipe-delimited, tab delimited, column delimited, ini file, proprietary binary serialization, locked down, non sharing, no service type world. It was the dark ages of data sharing. Hate on XML all you want, XML opened the doors.

Then comes XML, the executives and CTO magazines flooded with the term XML and large budgets signed on the word alone. But was it all hype or did it do something amazing? XML Amazing you say? With XML it was so simple it gave people no excuse not to open up information. A flawless victory on data nazi attitude. Is it the best, no, but it did what was necessary. We would not have the service based systems we have now of even JSON or other more micro formats at all if it weren’t for XML. Both HTML and XHTML and XML are all responsible (javascript as well and MIME) for delivering the simplistic base platform which all programmers can write to to instantly make their apps standard, the web and services that live on them.

After XML… Yes, XML did the amazing… It freed data into services. The web was also instrumental in this effort itself but when systems started working so closely together the exchange and mapping of data quickly became troublesome. Before web services emerged, client/server, remoting, RPC or other more closely coupled communication connections ruled the day. XML with web services helped to push the service model in addition to other technologies such as SOAP bloated but XMLRPC, REST, JSON, have emerged in stronger force or late because they are better iterations and less enterprise-y and simpler and more compact but I still believe that XML was in large part a tool that made data so simple to share that the capabilities and costs came down when people wanted to expose this data.

XML deserves to be a baseplane technology and is used where appropriate in baseplane tools and toolkits.

XML is recently 10 years old, seems like the average for standards to truly take hold and influence. Same with CSS, XHTML, the DHTML that later became AJAX and javascript kits of today. They are all stepping platforms.

baseplane: In software development, a set of tools, systems, patterns and designs that allow a system to be easily transferred to many back ends, services or front ends using standards, patterns and ideally non platform specific tools. Typically the baseplane includes both open standards and market standards based on industry usage. A baseplane may also be a way to commonize output to allow better designer and developer production.

A baseplane is a word I termed to describe the current, most technologically advanced set of tools and patterns that are usable across all major programming platforms.

The idea is that a company or programmer is sometimes held hostage by a particular platform or developer mindshare. People loving a good competition boast about platforms over other platforms and how one is better than the other. It is all fun and games but it doesn’t really give you the ‘best’ patterns and solutions. By patterns I mean general patterns not design patterns specifically for OO programming.

The ideal situation is where software is built semantically and from user perspectives with simplified enterprise entities and collections that perform top notch in a data store, no matter the platform or the language used to create it.

That is where true solutions begin, when companies and individuals can use the best platform or framework no matter their legion. A baseplane is a way to create systems that work well in many systems.

For instance JSON is a great example, this allows a layer to make back ends any language and then the front end any presentation from apps, to web to flash to whatever.

There will be more on this but the tools on this site all have this in mind.

Markets of many things have a period of innovation and experimentation through normal human trial and error, record response, adjust feedback loops. Over time, certain market standards emerge.

For social networks these market standards are profiles, friends, messages, blogs, pictures, and other custom handlers. OpenSocial has emerged late 2007 and is a great example of where after a period of time, a standard baseplane platform emerges and makes it another stepping stone to a further period of innovation and experimentation through normal human trial and error, record response, adjust feedback loops by combining and allowing a protocol to expose and consume data for OpenSocial. This could help many smaller or quickly developed communities get setup quickly, and it will also create social network giants that own the data. OpenSocial from Google was released after Microsoft signed with Facebook. Google came back with OpenSocial that got lots of attention and respec, also a move for Google to own the information and data standard. Then Facebook got forced to open up the apis to provide almost the same functionality, but no OpenSocial. Sometimes you need toolkits that will help to talk to both systems or many more pluggable ROM generation kits that allow you to integrate into these services such as blogging has the metaweblog apis and do most communication in a RESTful or XMLRPC based standard communication. It has allowed many blogging tools to integrate and help proliferate the platform and the companies involved in constructing it such as Wordpress (Automattic), MoveableType (SixApart), etc. There is great value to having open standards and market standards that allow the platform to flourish.

The baseplane code generation BOM (baseplane object mapper) has a ROM (rapid object mapper) both in development that can map your objects or existing applications to share along the OpenSocial standards (roadmaps and tests coming soon). We have many more ROMs that allow layers to be added to your code generation and custom manipulation of that work that have been built after 12+ years of experience in enterprise development. The great thing is the ROMs are custom, meaning that they follow your coding style and how you want things. None of this dictator like ORM systems, this is the reverse to that. They can integrate with existing code and they are not THE platform, they help you shape one though. More on this as the BOM is dropped.

The good news is more platforms are emerging, the bad news is there is much to learn and become an expert on.

It is a well known fact that humans are thrown into an abyss without much information given or input, they are simply thrown in and latch onto platforms and something to start relative understanding. This is why so many belief systems, points of view and general human pathing exists (more on pathing in future philosophy topics), when it is all trial and error it what works might be different for everybody.

So programmers or creators are the same, once they are thrown into the software world abyss, especially without theory or computer science, the programmer usually latches onto the closest, most low-entry-bar technology/platform/company/language for their use (whatever provides the cash money - you can apply this to any market/industry really).

Once that skill is mastered, they reach to others in the same space, they reach to some standards that mix with other platforms. Then, eventually, they reach the walls of the platform just like on the Truman Show (maybe a 2-5 year run), they realize it is a box not a baseplane. Once the abstraction of the platform to the solution and design inputs and outputs as a whole it becomes clear this is a power of 10 abstraction removed from a programmer stuck in platform-like religious battles, still latching on to a particular platform or language over general solution making. The programmer that rises above all this is the true master of the skill or talent if you want to call it, this is when they become an engineer.

Programmers are the ones doing the hard work to make it easier. As a programmer you have already committed to doing the hard work.

Consuming systems as the input and simplifying them as the output, the ultimate goal of a programmer.

Oh, and all programmers eventually become philosophers, it is a side effect that is spawned from an input of understanding users. Or in particular, how the human works and interacts with your system through HCI - Human Computer Interaction and the interface for which they do so.

Power of Ten Reference

[youtube]http://www.youtube.com/watch?v=BBsOeLcUARw[/youtube]

JSON is an amazing evolution to simplicity and one of the best new microformats of the 21st century. See in the early 90’s there were still many battles about formats, data and services really were non-existent. Sure there was some client/server based RPC and early remoting but it was more about protecting data and licensing fees to access data.

XML came along and many server platforms based on RFCs that made the web based on MIME and TCP/IP and many layers of the OSI. But XML really helped to simplify the output of data for many levels of developers and although bulky, it rooted out the data protectors and made it too easy to build systems that shared data. Formats like EDI and other tight textual formats survived the entire time but formats not destined to trade were also opened up with XML and spawned the service age.

The evolution to JSON a micro, object based and array based format is again so simple like XML and now so micro that it just is amazing to break it down to something that simple and have it spread (more…)