Integrating Legacy Systems

Recently we tried to integrate a legacy system via exposing a few mainframe programs via web services.

Exposing data as a web service does not make a web service.
Lets examine why:
a) Lack of re-usability - The mainframe programs return course grained data specific to a particular need it was developed for. This leads to having more data and more specific data than any other app would need to reuse the data. For example instead of a code or status returned it returns a message intended to be used by the consuming application. This data is virtually unusable to another application because most of the time you need the deciding code or status to perform logic later on.
b) Lack of performance - Instead of returning a code or status very verbose text is being returned. Now instead of a low overhead code or status you have a large message being transferred over the wire which may not be used. If this web service is consumed by another web service layer then transporting this around is going to be slow.
c) Design constraints - The mainframe program was designed for a particular use which they didn't take into account limiting the repetitive sections. They would return all 100 instances of a section of data. If only the first 5 had data they would return all 100. This adds great overhead on a web service. Also the data that comes from a mainframe program is not trimmed and has filler fields filled with whitespace.


If you don't have somebody regulating or asking the right questions then anything will become a web service which shouldn't be. I recommend a group of architects and/or tech leads making the decision of whether something should be a service.

Here are some guidelines/questions to ask yourself for web services:
1) Ask yourself or group why are we making this a web service? What are we hoping to accomplish?
a) When you go to a web service you are giving up performance for agility.
b) Make sure you're willing to deal or accept the trade off.
2) Does it meet any of this criteria?
a) Is this going to reusable by other applications? Which ones?
b) Is this something that crosses boundaries or something which I can't control? For example if you are exposing something to another business unit or exposing it for public use.
c) Is this enabling system integration for systems that were not able to integrate before.
d) Is this coarse grained enough for general use?
e) Is this data going to be suitable for multiple hops?

Building the Base - Data Integrity

Something I feel is far too often overlooked in software development is data integrity.
I would like to further define it as strong data typing.

The source of lack of strong data typing is several places.
1) Lack of proper column types in the database.
2) Reuse of typed columns to mean different things.
3) Programs/Stored Procedures that decrease the strength of data typing.
4) Flat file (fixed record formats)

Now I'm not a DBA or anything like that.
But one thing I've noticed when developing JSF applications is lack of data integrity seems to bubble it's way up into the presentation layer.

It should be dealt with at the source to provide a better base for an presentation/gui layer.

1) Lack of proper column types in the database.
If you are using clobs, blobs, text, varchar, or whatever flavor of large catch-me-all for typed data please stop. One symptom is that all your columns seem to have the same type. Please use date, int, time, timestamp, bit, numeric, float, decimal, bigint, tinyint.

2) Reuse of typed columns to mean different things.
Use of low values or high values to mean something:
99.99, -1, or any date like January 1st, 1900.
I think almost any database or language has null built in for this reason.
I'll see logic if 0.00 or 99.99 then display N/A so why don't we just define another column with a one character value and a domain table.
So instead of this:
Effective Date: Jan 1st, 1900
We do this:
Effective Date: null Effective Status: 'P'
where there would be a Domain Table with things like 'P' - Pending, 'T' - Terminated, 'I' - Invalid and 'U' - Under Review


3) Programs/Stored Procedures that decrease the strength of data typing.
If you read in two strongly typed and return a lesser typed one this would be a smell. For example if you take two Numbers in and generate a String number result and N/A result sometimes.

If you take two specific things and return a generic one then that is a smell.
For example if you take name fields and put them together to display. Well this is all good but what if you then want to pre-populate a name into a form. How do you parse a complex name which may or may not have middle initials and a suffix?

4) Flat file (fixed record formats)
This is the root of all evil because there are no rules as to what goes where as long as it fits.
I propose that you establish a contract with the generator about the types if possible.
Don't leave it in flat file format, extract it into a strongly typed database and throw out invalid data. It is better to throw out or reject rows than to have it in your database.

No Fluff Just Stuff

This is the second year I've attended the No Fluff Just Stuff conference.

If you've never experienced the conference and you do any kind of software development that involves/runs on the Java VM (Java EE, Java SE, Groovy, Grails, etc) I strongly encourage you to go.

Even if you don't do any Java related software development but you work in Quality Assurance/Testing or Business Analyst/Requirements Development there are beneficial courses for you too. Some of these courses seem to be more geared towards this group but are attended mostly by developers. This seems to stress the notion of one software developer to change the world. One thing that comes from some of these sessions is the notion of collaboration. So instead of one person in one group (Software Development in my case) I'd like to encourage people from other groups (Quality Assurance/Testing or Business Analyst/Requirements Development) to attend also. This way we can change the world together!

The only thing I can say in any way negative about this conference is they seem to have a lot of things that are new and shiny new things like Groovy, Grails, Agile, and GWT. I work in a large behemoth of a company that is a slow moving, slow to adopt, vendor locked, internal standards driven. These things seem to be something that can be implemented more in small shops or don't have resistance to change (or regulations). Even this being said I still thing attending some of these are good for me because you can extrapolate some of the tips for your organization even if it really is an apples vs oranges environment.

I really enjoyed some of the concurrency, architecture and scaling, SOA, and Agile sessions. Last year I liked some of the continuous build, testing, and hacking sessions. I find some of the Facelets, JSF, Spring, Hibernate courses rather introductory but for some might be invaluable.

One thing (it's hard to name everything) I really like about the conference is that it is small enough you can ask questions and have a small group feel. A lot of the sessions are interactive and make good points.

You really get a lot of books recommendations from this conference some related more to software development philosophy rather than really technical books.

The keynote speech is greatly entertaining, funny, and informative.

This conference is also big on open source and you don't find any real vendor driven presentations.

JSP vs Facelets

One of the saddening aspects of JSF is the rendering of HTML along with JSF components in JSP pages.
For example

<div><div>Some Header</div>
<ul>
<li><h:outputText value="list item 1">
</h:outputText></li><li><h:outputText value="list item 2">
</h:outputText></li></ul>

will actually render as

list item 1list item 2
Some Header
<ul><li>
</li><li>
</li></ul>

Alright now the problem is that the HTML is written out right away and the JSF components are rendered at a different time.

Another situation is when you have HTML before or after a JSF Component.

<h:panelGroup>
<p>some markup</p>
<h:outputText value="some text"/>
<p>some more markup</p>
</h:panelGroup>

will render as

some text
some markup
some more markup

Now if you ask me since the default view is JSP for JSF they should have taken this into account.

They should key off the include of JSF tags in a page to parse the JSP differently.
Instead of rendering it like they do they should gather the HTML and attach it to the view in a verbatim component or some other kind of ui output component in the appropriate place in the JSF component tree. This should be possible because everything should be enclosed in a <f:view> tag.

Either that or you have the custom jsf tags be body aware which I assume they are and output the html prior to rendering their component/child component and then render html after the component.

I'm not sure which way is best or where this would take place but it seems very doable.

Now some people say Facelets is the answer and dump JSP all together.
I agree this is a way to fix it but this doesn't let you reuse any JSP tags that are non-JSF.
In the real world there are lots of JSP tags that are not JSF that still need to be reusable.
Also there was a reason for JSP pages in the first place and I'm not sure facelets fully fills that gap.

The real answer should be to fix JSP complation, JSP Rendering, or JSF tags where ever this should take place.

Whoever was responsible for the JSF spec should have seen to this in the first 1.0 release.

Tiles Killed Convention - Facelets Templating

I recently went to a developers conference and listened to a rapid application development with JSF session by David Geary. I found the session very helpful and informative but something keeps bothering me every time somebody talks about templating as he did with Facelets.

Every time I hear about templating I think it should be easier and more reuseable. I started out developing Struts applications which use Tiles. I then moved on to JSF and Tiles. The nice thing about Tiles is "it works". Sometimes just working is not enough and a little convention goes a long ways.

Recently I've been thinking about the convention over configuration mentality that RoR uses. Also I've come upon a great templating package called Sitemesh. What I love about Sitemesh is you can define a decorator and decorate multiple pages by a matching URL pattern. This is a GREAT idea. Using Tiles I've always had to create a definition for every page in a Tiles definition file, a JSP that inserts the definition, and the page that actually has the content. Well that's three files for each page of content the user sees. Now the problem I have with this is the Tiles definition for the page and the tiles insert JSP is basically repeated for every page. Sure you can define a base Tiles definition and extend it but it tends to be mostly repetitive. You can also define your tiles definition in the JSP page which is just as messy and repetitive.

Let me get back to Facelets. Facelets is a wonderful idea to use the html view and bind to components. The problem is when you get to the templating part it goes back to the Tiles mentality again. You have a page which defines a composition and a page which contains the content. The page with the composition is repeated for every content page and in most cases the exact same composition definition. This is where the convention of Sitemesh is nice and could come in handy. It would be nice if you could define a Facelets composition for a URL path and apply it to all pages in a directory. That's a heck of a start for reuse and convention over configuration. Then you can just configure the exceptions to the case which should be rare.

Sidenote: You can use JSF components in a Sitemesh decorator (http://jira.opensymphony.com/browse/SIM-201).

There are a couple of positive benefits to applying templates based on a URL path that are a side effect. I've been on too many projects where everything is thrown into the base directory of a web application or relatively few directories. This gets out of hand quickly and not good structure. There are also security packages like Acegi that will apply security based on the URL path. So if you have your files in an appropriate file structure it should be easy to find pages, apply templates, and apply security.

(If you don't think it can be done check this out: http://jira.opensymphony.com/browse/SIM-223)

This brings up another annoyance of JSF. JSF has no notion of subdirectories built in. With Struts you could define config files for each URL path and then you didn't need to put the subdirectory paths for the JSP pages in a configuration file. JSF should add this in and promote appropriate directory structure.

As a closing note if you're doing something multiple times and not much changes maybe you should refactor it out. If you are doing this in programming maybe you would extract the commonalities out into a method passing in the parameters and then the method can be reused. In the case of a templating package you should refactor the configuration out of the JSP page or Faclets xhtml view and move it into the URL path with a configuring Servlet or Filter.

Optimistic Locking in Rails with Active Record for Free...almost

Like other persistence frameworks, ActiveRecord helps us with concurrency issues. This is actually fairly simple, but in general the documentation I have found on this subject has been poor. I will try to sum this up very quickly.

1) Add lock_version

For each table where concurrency is a concern a lock_version column must be added; Rails is then supposed to perform some magic behind the scenes...

In your migration this should look as follows.
table_name.column :lock_version, :integer, :default=>0

Alternatively, if you already have a versioning column in your database you can inform rails of your column by adding the line "set_locking_column 'your_versioning_column_name'" to the associated model class.

2) Catch StaleObjectError

Now that you have added the lock_version column you get concurrency for free. Whenever an attempt occurs to save over the same record version an ActiveRecord::StaleObjectError will be raised.

So now you can update your controllers to catch StaleObjectErrors and handle these however you wish...

This would look something like this.

def some_updating_method(an_object)
an_object.update_attributes(params[:an_object])
rescue ActiveRecord::StaleObjectError => e
flash[:notice] = "This record changed while you were editing it. Please try again."
#maybe perform a redirect here...
end

3) But Wait! Update methods aren't actually passed objects that's too much overhead!

How observant you are. This is where most of the example tend to fall short. In fact, try implementing your update method exactly like Recipe 3.18 from the Rails Cookbook and see what happens...

Generally an update method actually looks more like this.

def update
my_object = MyObject.find(params[id])
my_object.update_attributes(params[:my_object])
end

While this will stop to users from updating at the exact same time this isn't exactly what we want in many cases. More often than not we want to prevent two users from editing over each other without knowing.

This is easily fixed with the following...

def edit
my_object = MyObject.find(params[id])
session[lock_version] = my_object.lock_version
# whatever we want edit to do
# ...
# ...
# call update...
end

Now we update update (har har) to detect the concurrency issue...

def update
if my_object.lock_version != session[:lock_version]
raise ActiveRecord::StaleObjectError
end
my_object = MyObject.find(params[id])
my_object.update_attributes(params[:my_object])
rescue ActiveRecord::StaleObjectError => e
flash[:notice] = "This record changed while you were editing it. Please try again."
#maybe perform a redirect here...
end

Hopefully this helps someone out. If you have a better/cleaner way of doing this, let me know.

Drop all tables in MySQL

From time to time I would like to be able to drop all of the tables in my MySQL database without actually dropping the database. This can be painful if there are many tables in the database. Recently I came across this little nugget that simplifies the process.

mysqldump -u[username] -p[password] --add-drop-table --no-data [database] | grep ^DROP | mysql -u[username] -p[password] [database]

Voila! All your tables are gone!