Recently we tried to integrate a legacy system via exposing a few mainframe programs via web services.
Exposing data as a web service does not make a web service.
Lets examine why:
a) Lack of re-usability - The mainframe programs return course grained data specific to a particular need it was developed for. This leads to having more data and more specific data than any other app would need to reuse the data. For example instead of a code or status returned it returns a message intended to be used by the consuming application. This data is virtually unusable to another application because most of the time you need the deciding code or status to perform logic later on.
b) Lack of performance - Instead of returning a code or status very verbose text is being returned. Now instead of a low overhead code or status you have a large message being transferred over the wire which may not be used. If this web service is consumed by another web service layer then transporting this around is going to be slow.
c) Design constraints - The mainframe program was designed for a particular use which they didn't take into account limiting the repetitive sections. They would return all 100 instances of a section of data. If only the first 5 had data they would return all 100. This adds great overhead on a web service. Also the data that comes from a mainframe program is not trimmed and has filler fields filled with whitespace.
If you don't have somebody regulating or asking the right questions then anything will become a web service which shouldn't be. I recommend a group of architects and/or tech leads making the decision of whether something should be a service.
Here are some guidelines/questions to ask yourself for web services:
1) Ask yourself or group why are we making this a web service? What are we hoping to accomplish?
a) When you go to a web service you are giving up performance for agility.
b) Make sure you're willing to deal or accept the trade off.
2) Does it meet any of this criteria?
a) Is this going to reusable by other applications? Which ones?
b) Is this something that crosses boundaries or something which I can't control? For example if you are exposing something to another business unit or exposing it for public use.
c) Is this enabling system integration for systems that were not able to integrate before.
d) Is this coarse grained enough for general use?
e) Is this data going to be suitable for multiple hops?
Thursday, August 21, 2008
Saturday, August 16, 2008
Building the Base - Data Integrity
Something I feel is far too often overlooked in software development is data integrity.
I would like to further define it as strong data typing.
The source of lack of strong data typing is several places.
1) Lack of proper column types in the database.
2) Reuse of typed columns to mean different things.
3) Programs/Stored Procedures that decrease the strength of data typing.
4) Flat file (fixed record formats)
Now I'm not a DBA or anything like that.
But one thing I've noticed when developing JSF applications is lack of data integrity seems to bubble it's way up into the presentation layer.
It should be dealt with at the source to provide a better base for an presentation/gui layer.
1) Lack of proper column types in the database.
If you are using clobs, blobs, text, varchar, or whatever flavor of large catch-me-all for typed data please stop. One symptom is that all your columns seem to have the same type. Please use date, int, time, timestamp, bit, numeric, float, decimal, bigint, tinyint.
2) Reuse of typed columns to mean different things.
Use of low values or high values to mean something:
99.99, -1, or any date like January 1st, 1900.
I think almost any database or language has null built in for this reason.
I'll see logic if 0.00 or 99.99 then display N/A so why don't we just define another column with a one character value and a domain table.
So instead of this:
Effective Date: Jan 1st, 1900
We do this:
Effective Date: null Effective Status: 'P'
where there would be a Domain Table with things like 'P' - Pending, 'T' - Terminated, 'I' - Invalid and 'U' - Under Review
3) Programs/Stored Procedures that decrease the strength of data typing.
If you read in two strongly typed and return a lesser typed one this would be a smell. For example if you take two Numbers in and generate a String number result and N/A result sometimes.
If you take two specific things and return a generic one then that is a smell.
For example if you take name fields and put them together to display. Well this is all good but what if you then want to pre-populate a name into a form. How do you parse a complex name which may or may not have middle initials and a suffix?
4) Flat file (fixed record formats)
This is the root of all evil because there are no rules as to what goes where as long as it fits.
I propose that you establish a contract with the generator about the types if possible.
Don't leave it in flat file format, extract it into a strongly typed database and throw out invalid data. It is better to throw out or reject rows than to have it in your database.
I would like to further define it as strong data typing.
The source of lack of strong data typing is several places.
1) Lack of proper column types in the database.
2) Reuse of typed columns to mean different things.
3) Programs/Stored Procedures that decrease the strength of data typing.
4) Flat file (fixed record formats)
Now I'm not a DBA or anything like that.
But one thing I've noticed when developing JSF applications is lack of data integrity seems to bubble it's way up into the presentation layer.
It should be dealt with at the source to provide a better base for an presentation/gui layer.
1) Lack of proper column types in the database.
If you are using clobs, blobs, text, varchar, or whatever flavor of large catch-me-all for typed data please stop. One symptom is that all your columns seem to have the same type. Please use date, int, time, timestamp, bit, numeric, float, decimal, bigint, tinyint.
2) Reuse of typed columns to mean different things.
Use of low values or high values to mean something:
99.99, -1, or any date like January 1st, 1900.
I think almost any database or language has null built in for this reason.
I'll see logic if 0.00 or 99.99 then display N/A so why don't we just define another column with a one character value and a domain table.
So instead of this:
Effective Date: Jan 1st, 1900
We do this:
Effective Date: null Effective Status: 'P'
where there would be a Domain Table with things like 'P' - Pending, 'T' - Terminated, 'I' - Invalid and 'U' - Under Review
3) Programs/Stored Procedures that decrease the strength of data typing.
If you read in two strongly typed and return a lesser typed one this would be a smell. For example if you take two Numbers in and generate a String number result and N/A result sometimes.
If you take two specific things and return a generic one then that is a smell.
For example if you take name fields and put them together to display. Well this is all good but what if you then want to pre-populate a name into a form. How do you parse a complex name which may or may not have middle initials and a suffix?
4) Flat file (fixed record formats)
This is the root of all evil because there are no rules as to what goes where as long as it fits.
I propose that you establish a contract with the generator about the types if possible.
Don't leave it in flat file format, extract it into a strongly typed database and throw out invalid data. It is better to throw out or reject rows than to have it in your database.
Sunday, August 3, 2008
No Fluff Just Stuff
This is the second year I've attended the No Fluff Just Stuff conference.
If you've never experienced the conference and you do any kind of software development that involves/runs on the Java VM (Java EE, Java SE, Groovy, Grails, etc) I strongly encourage you to go.
Even if you don't do any Java related software development but you work in Quality Assurance/Testing or Business Analyst/Requirements Development there are beneficial courses for you too. Some of these courses seem to be more geared towards this group but are attended mostly by developers. This seems to stress the notion of one software developer to change the world. One thing that comes from some of these sessions is the notion of collaboration. So instead of one person in one group (Software Development in my case) I'd like to encourage people from other groups (Quality Assurance/Testing or Business Analyst/Requirements Development) to attend also. This way we can change the world together!
The only thing I can say in any way negative about this conference is they seem to have a lot of things that are new and shiny new things like Groovy, Grails, Agile, and GWT. I work in a large behemoth of a company that is a slow moving, slow to adopt, vendor locked, internal standards driven. These things seem to be something that can be implemented more in small shops or don't have resistance to change (or regulations). Even this being said I still thing attending some of these are good for me because you can extrapolate some of the tips for your organization even if it really is an apples vs oranges environment.
I really enjoyed some of the concurrency, architecture and scaling, SOA, and Agile sessions. Last year I liked some of the continuous build, testing, and hacking sessions. I find some of the Facelets, JSF, Spring, Hibernate courses rather introductory but for some might be invaluable.
One thing (it's hard to name everything) I really like about the conference is that it is small enough you can ask questions and have a small group feel. A lot of the sessions are interactive and make good points.
You really get a lot of books recommendations from this conference some related more to software development philosophy rather than really technical books.
The keynote speech is greatly entertaining, funny, and informative.
This conference is also big on open source and you don't find any real vendor driven presentations.
If you've never experienced the conference and you do any kind of software development that involves/runs on the Java VM (Java EE, Java SE, Groovy, Grails, etc) I strongly encourage you to go.
Even if you don't do any Java related software development but you work in Quality Assurance/Testing or Business Analyst/Requirements Development there are beneficial courses for you too. Some of these courses seem to be more geared towards this group but are attended mostly by developers. This seems to stress the notion of one software developer to change the world. One thing that comes from some of these sessions is the notion of collaboration. So instead of one person in one group (Software Development in my case) I'd like to encourage people from other groups (Quality Assurance/Testing or Business Analyst/Requirements Development) to attend also. This way we can change the world together!
The only thing I can say in any way negative about this conference is they seem to have a lot of things that are new and shiny new things like Groovy, Grails, Agile, and GWT. I work in a large behemoth of a company that is a slow moving, slow to adopt, vendor locked, internal standards driven. These things seem to be something that can be implemented more in small shops or don't have resistance to change (or regulations). Even this being said I still thing attending some of these are good for me because you can extrapolate some of the tips for your organization even if it really is an apples vs oranges environment.
I really enjoyed some of the concurrency, architecture and scaling, SOA, and Agile sessions. Last year I liked some of the continuous build, testing, and hacking sessions. I find some of the Facelets, JSF, Spring, Hibernate courses rather introductory but for some might be invaluable.
One thing (it's hard to name everything) I really like about the conference is that it is small enough you can ask questions and have a small group feel. A lot of the sessions are interactive and make good points.
You really get a lot of books recommendations from this conference some related more to software development philosophy rather than really technical books.
The keynote speech is greatly entertaining, funny, and informative.
This conference is also big on open source and you don't find any real vendor driven presentations.
Subscribe to:
Posts (Atom)