It is a war out there, a war of attrition. The Internet market is saturated with Internet service providers (ISPs), value-added resellers (VARs), marketing firms, advertising agencies, consulting firms, software companies, Web-development firms, public relations agencies, graphics design firms, New Media companies, and Joe (Frank’s friend that knows computers) that all compete for Web-development projects.
Just about anyone can create a Web site. Some companies do a better job than others, but a basic site isn’t that hard to throw together. Competition is fierce for standard Web development, but as complexity increases, competition decreases.
For those of you who crave challenge in your projects and would like to increase your value or decrease your competition, you may want to consider joining an elite group of professionals known as Web-based systems integrators.
Introducing “The Few, The Proud, The Web-Based Systems Integrators.” If you are ready to take Internet programming to the next level, continue reading, For the remainder of this article you are no longer a civilian. Welcome to Basic Training.
Wake up! I am Drill Sargent Tanner. My job is to teach you the skills you need to survive as a Web-based systems integrator.
When you complete Basic Training (finish this article) you will:
Know what Web-based systems integration means,
Understand your role as a Web-based systems integrator,
Be able to plan your attack,
Understand how to evaluate the terrain,
Know how to choose your weapons for the situation,
Be able to engage in hand-to-hand combat,
Recognize booby traps and how to disarm them, and
Be ready for mission-specific training.
What is Web-Based Systems Integration?
Web-based systems integration is the art of combining multiple systems (including Legacy systems and proprietary software applications) into a new system that is accessible through a Web browser.
Your job as a Web-based systems integrator will be to create these Web-based systems using a variety of technologies and tools. No two systems are going to be the same, so it is important that you diversify your knowledge and skills as technology changes. Web-based systems integrators typically have titles such as software engineer, consultant, senior programmer, etc.
Regardless of your title, you should have a technical background and be familiar with multiple platforms, multiple programming languages, various Internet technologies, and the tools of the trade. You should also have the ability to lead others and communicate with clients in nontechnical terms. Web-based systems integrators are big-picture people with the skills to perform technical magic and make it look simple.
If you don’t have all of the skills listed above, you have some catching up to do. But don’t let that get you down. The best way to get the experience you need is to work in a cross-functional programming team where the combined knowledge and expertise of all the team members meets or exceeds the requirements of the project.
There are many ways to connect Legacy systems to the Internet/Intranet. Some of these include screen scraping, direct file/database connections, replicating Legacy data, interfacing with CICS, and more. Each of these methods has its advantages and disadvantages. Choosing the right method requires detailed knowledge of the environment, client goals, platform mix, requirements, and situation. In this article, I will be highlighting the direct file/database connection method.
Details about one of my recent missions will give you a better understanding of what you can look forward to.
Situation and Condition
The client issued a Request for Proposal (RFP) to several vendors with technical skills and a history of success with high-end Web development.
They wanted a system to pull property data from an AS/400 and display images of the property documents over the Internet. Sounds simple enough, doesn’t it? However, after visiting with the potential client, I uncovered some more details.
The original system was written in RPG/400 and contained 45 tables and 27 views in a DB2 database. The images were contained within a Proprietary Document Management System (PDMS) and were stored on an optical drive in TIF format.
The PDMS ran on a Windows 95 machine and used an SQL Server 6.5 database on a separate Windows NT machine to store information about the images and their location on the optical drive.
The client’s existing Web site ran on a Windows NT 4.0 server with Internet Information Server 4.0, and they wanted this new system to be accessible through their Web site and run on the same machine.
We had one week to come up with a solution and present the client with a bid for the development. Let me break this project down into individual components:
Create Web pages to replace screens on the AS/400.
Establish a connection with the AS/400 DB2 database.
Pull data from the AS/400 directly, but process the data within our new system.
Re-create the business rules underlying the existing system.
Interface with SQL Server to determine where specific images are on the optical drive.
Display the TIF images through a Web browser.
Now that you understand the components of the system, let’s see what kind of solution we put together over the next few days.
How this section relates to you:
It is your job to make sure you know what the client is asking for and break it down into manageable components. Don’t be afraid to ask stupid questions. Even simple issues can dramatically alter your programming direction. You will have a much better vision of the solution if you start with a clear picture of the existing system, issues, and client needs. Keep in mind that the word “client” throughout this document may refer to your customer or your own company, if the project is internal.
Strategy
My initial thoughts on the solution were as follows:
Designing the screens with the client’s input, and guidance from the programming department would be a piece of cake for our production staff. So my thoughts on the solution focused almost exclusively on the technological aspects of the system.
I determined that we would use an ODBC connection to the DB2 database and use the Internet to search for available drivers. I made a note to follow up with the client to determine if any of the drivers I found were in their possession. If so, the driver they had would be my default choice.
Rebuilding the business rules seemed straightforward. I felt that I had a fairly good understanding of what was happening behind the scenes of the existing system. Having the actual source code would guarantee correct functionality, but we could manage without it. This was not a major stumbling block.
Interfacing with SQL Server would be simple enough. Since we knew that there was a relationship between the DB2 database and the SQL Server database, there was no need to get any more detailed in this area. As long as we could determine the location where a specific image resided on the optical drive, we were in good shape.
Lastly, we needed to display those TIF images through a Web browser. My initial thought was to use a Java-based TIF viewer to display the images, since Web browsers can only view GIF and JPG images natively. In order to have the Java viewer display the TIF image, I would need to pass a path to a specific image. Since I wasn’t sure how the optical drive was formatted, I needed to clarify this with the client.
I called the client and asked if they could view a specific TIF file by opening Windows Explorer on their computer and displaying available images on the optical drive. The client said, “Yes” to this question. Great. This meant that I could have the Java viewer open a TIF image directly from the optical drive and display that image through the browser using Java.
I also asked if they had any of the ODBC drivers I mentioned earlier. They said that they had Client Access. So Client Access it is. If the client didn’t have any of the drivers mentioned, I would have chosen the driver that required the least work on our part and would handle the clients’ long-term needs for the system.
Now, should we develop the Java-based TIF viewer internally or look for a third-party product? It is less expensive and less time-consuming to use a good third-party product if one exists, so I searched the Internet for similar products. I found several TIF Java development kits and finally came across one product that looked like it would work for us. I downloaded the product and tested it. It was developed by a German company and worked great. In fact, it was specifically designed to work with document management systems.
At this point, I was ready to present my solution to the client. It was very important to let the client see that we understood the system and related issues, and that our solution was a total solution. The sales manager and I got together to create a proposal that explained our solution in a way that the client would understand. We hand-delivered the proposal and went through it with the client so we could answer any questions.
The client gave our sales manager a call a few days later to inform us we were selected as the vendor of choice. When asked why the client chose us over the other firms, we were told that we were the only vendor to present an actual solution. All other potential vendors presented only partial solutions or suggested a planning/needs assessment phase-in which they would later determine if the project were feasible.
How this section relates to you:
Once you have determined the primary components of the system, you need to come up with a high-level solution for putting the system together and dealing with forecasted problems. It is in your best interest to have a good overall understanding of the existing system as well as a detailed understanding of the solution you are presenting. The client will question you on both of these.
Now let’s meet the team that made it all happen.
Your Platoon
Don’t think the entire weight of a project rests on your shoulders alone. Web-based systems integrators work within teams of professionals with complementary skills.
A Web-based project requires planning and coordination, HTML, writing and editing, graphic design, and programming. If you try to do it all yourself, you will end up with mediocre results and the project will take 2 to 3 times longer. Furthermore, your time is too valuable technically to be spending any of it on nontechnical tasks.
My team was made up of the following professionals:
Production Coordinator: The production coordinator met with the client to iron out the details of the screens and determined how the Web-based screens could make data entry easier for the end user. She created a storyboard that based the navigational structure of the system on the existing application. Then she delegated various tasks to the information architects and graphic designers.
Graphic Designer: The client had its own graphic designer, and we had to work with the established design. Unfortunately, the client is not an expert in graphic design for the Internet, but our graphic designer did a reasonable job in setting up a layout that would be conducive to programming, while keeping the client’s existing theme intact. Clients typically like to sink their money into the functional portions of the site that their staff are not well suited to develop, and spend less money on interface and content development. Whenever possible, try to get the client to invest in the interface and content portions of the system, and they will be happier in the long run. Trying to save them money up front is fine, but you should inform them that adding those components later will cost them more.
Information Architects: Two of our information architects put the various screens together using the mock-ups prepared by our graphic designer. Screens were laid out on paper first, with the guidance of our programming department.
Programmers: Two of our programmers contributed to the back-end functionality of the system. An SQL Server was initially used to mock the data on the AS/400 for development purposes. Site-programming standards were developed and followed by all programmers for consistency purposes. Programmers consulted with me when questions arose.
Senior Programmer: I planned and designed the technical portion of the Web-based solution. I delegated the programming tasks and reviewed progress throughout the development cycle. I kept the client updated on a weekly basis, or as special circumstances arose. I met with the original vendor to determine special functionality requirements and inquired about Year 2000 issues. I created the specifications that the programmers used to build their efforts. I made regular site visits to compare functionality on the new system against the existing system.
Now let’s take a look at the environment we were working with.
How this section relates to you:
As I stated at the top of this section, you are not developing the entire system yourself, unless you happen to be an expert in graphic design, HTML, writing and editing, programming, and project management. A solid Web-based project requires all of these disciplines. You have to be a team player. The benefit of using a team approach to the project is that the project will be completed in less time, and the quality will be higher.
Terrain
"The Terrain" image illustrates what our environment looked like. We were dealing with four servers and three data sources. A fire wall separated the Web server from the other servers, and we were working with an Ethernet network. The network was configured with TCP/IP, which made our job a little easier.
The client was using Internet Information Server 4.0 and SQL Server 6.5, which made Active Server Pages an easy choice for the development of this system. Knowing your environment is very important, but you will need a lot more information up front to provide your team with a solid direction. That is where the map and compass section of this article comes in.
How this section relates to you:
It is important to know the constraints you will be dealing with and what systems you will be integrating. Your environment represents a balance between the tools, languages, hardware, and software you have to work with, and the constraints you face. It is almost always best to leverage existing resources whenever possible. For example, if the client is already using Oracle and you need to add an additional database, use Oracle. The client already has the licenses and the staff already feels comfortable with Oracle. Leveraging existing resources also costs the client less.
Map and Compass
Whenever possible, it is best to have resources, such as the original developers and end users, available to answer any questions you may have before and during development. You will also want to get your hands on all available system(s) documentation.
The vendor of the existing system graciously provided us with the source code and walked me through a few of the programs so I could get up to speed quickly with their naming conventions and programming style. I left with a huge stack of RPG/400 code, screen specifications, and database documentation. The client also provided us with the user manual. The manual aided us in understanding the system from the end –user’s perspective.
The original vendor explained that they understood what we were doing, but not how we were going to make it happen. How would we replace the function keys that pulled up subscreens? How would we maintain state and scroll back and forth through large record sets? I assured them that we had many alternatives and that I would be happy to demonstrate the product once it was completed.
While we were busy obtaining as much technical information as possible, our production coordinator was meeting with the primary end users. She laid out the screens on paper before we ever attempted to do the actual work. Her goal was to make sure that the layout and navigation scheme of the system would be exactly what the end users wanted. Some of the screens on the AS/400 were split into multiple screens in the new system, while other screens were consolidated into a single screen. Since we were using different technology, it only makes sense that we may be able to take advantage of our modern tools and techniques to make the system easier to use.
Once the system was laid out, the graphic designers created mock-up screens with the guidance of the programming department for technical feasibility. The screens were approved by the client with only minor modifications and passed on to the information architects for assembly.
Our information architects used FrontPage98 as their primary tool for creating the screens and used the mock-ups provided by the graphic designers as templates to maintain both color schemes and page structure.
When all of the screens had been built, they were handed off to the programming department. This is when all the fun began.
How this section relates to you:
In the development of a new system, it is best to arm yourself with as much documentation and knowledge about the existing system(s) as possible. Your client will want you to understand the existing system thoroughly, so it is in their best interest to supply you with the documentation you request. Entity-relationship diagrams, data-flow diagrams, data dictionaries, source code, and user manuals are a good start. You can also interview end users before development starts.
Weapons
Before I get too involved in what the programming department did, I want to talk about the tools that were used.
Keep in mind that we chose these tools for this project, but we could have chosen other tools and accomplished the same results. The goal here is just to introduce you to some of the tools, languages, and technologies that were involved.
Our graphic designers used PhotoShop and FrontPage98 as their primary tools.
Our information architects used FrontPage98 almost exclusively, as well as Notepad, once programming was embedded in the HTML.
Our programmers used Visual Interdev, Visual Basic, and Visual C++. We used SQL Server to mimic the DB2 database until it was time to move the system to the client’s location for final testing.
We also read thousands of lines of RPG code that made up the business rules of the new system. Most of the development was done using Active Server Pages, but as you will see in the section labeled "Obstacle Course," unanticipated events required the use of C++ programming.
How this section relates to you:
You must choose the tools, languages, technologies, hardware, and software needed to implement the solution. You won’t be able to use the same combination every time because you are bound by the constraints of your environment and the needs of the client. Keep in mind that your selection in this area will impact the development of the system.
Hand-to-Hand Combat
When providing a Web-based alternative to a Legacy system, it is important to consider how you will substitute function keys, subscreens, persistent state, paging up and down, and other Legacy system features.
For those of you with little or no Legacy system experience, I will explain these concepts further.
The system we were replacing had dynamic function keys that appeared at the bottom of a screen to indicate what options were available to the user. For instance, by pressing F7 a user could access legal information regarding the current record.
Subscreens are pop-up lists of valid values for a specific field. For instance, pressing F4 on the parcel number field would pull up a miniature window with valid parcel numbers. The user would put an X in front of the parcel number they wanted to select, and then press enter. The selected parcel number would then be entered in the parcel number field.
Persistent state is a tricky issue. When accessing a Legacy system, the application establishes a constant connection with the user until the user logs off or exits the system. All current information is held in memory. In contrast, the HTTP protocol is stateless. The connection between a browser and a server is severed after each request/response.
When users perform a search on a Legacy system, they are typically presented with the first of several screens of records from which to select. They scroll through the records by paging up and down through the record set. They have the ability to mark records on one screen, scroll to other screens, and mark additional records for viewing. Furthermore, the user can revisit previous screens to change their selections. Once they are satisfied, they choose to view the selected records one at a time.
We need to provide equivalent or better functionality than the users are accustomed to in the Legacy world through Web-based systems. That is what this section is all about.
We replaced dynamic function keys with dynamic hyperlinks. In other words, if a record had corresponding legal information, we provided a link to that legal information. If the record did not have corresponding legal information, we didn’t present the link.
We replaced subscreens with either drop-down lists or a new window. If a field had a small number (i.e., less than 100) of possible values, we pulled the information from the database to populate a drop-down menu. We then ordered the list alphabetically so users could type the first letter of the value they are looking for and immediately position their cursor near that value.
If the number of values was too large for a drop-down list, we could pull up a new window to allow the user to perform a search for the value. For instance, we could allow the user to type in a partial address and display a list of matching results. The user could either cut-and-paste the correct value into the previous window, or if we wanted to get fancy, we could have developed code to pass the information back to the initial field once the user selected the valid record.
The persistent state issue can be solved a few different ways. We chose to use session variables in our application. Session variables take advantage of cookies in a way that hides the details from the developer. It is a good method to use if you know your audience won’t mind accepting cookies, or you can use something similar to the cookie munger – an ISAPI filter used to handle cookies on the server side.
We used session variables to store arrays of record sets to handle the issue of paging up and down. The implementation was a bit tricky and certainly system specific, but it is a good way to get around this issue. The screen print of one of the system screens gives you an idea of our implementation.
How this section relates to you:
There are many features of a Legacy system that users become comfortable with, and you have to provide suitable alternatives to these features. The goal is to improve the system by making it intuitive for the end user. Year 2000 issuesare definitely important. In fact, many companies choose to convert their old Legacy systems to Web-based systems since they are littered with Y2K problems. If you are moving the data to another platform, you will have more control of Y2K issues. (Year 2000 is a subject for a different article.)
Obstacle Course
No matter how much planning you put into a system, it seems that unforeseen obstacles creep up where you least expect.
The first obstacle dealt with the connection to the AS/400. We set it up properly; it just didn’t work. We knew it was a security issue, but weren’t exactly sure how to fix it. I informed the client of the problem and suggested that we contact IBM for a solution. My goal was to minimize my time. I didn’t want to bill the client for many hours of my time when IBM could give me the solution over a single phone call. The client was not initially convinced that IBM would be able to provide a solution over the phone and suggested that I meet with another vendor that installed Client Access on the Web server and happened to be a partner with IBM.
I met with the vendor and knew immediately that the vendor did not know what they were talking about, since I was having to explain simple concepts to them. After two hours, the vendor wished me luck. I got authorization to call IBM and had the solution I was looking for in less than 15 minutes. It seems that there is a configuration tool that comes with Client Access that allows guest users to access the AS/400.
The next obstacle involved the imaging piece. It seemed that the client was correct in stating that the document management system stored the images as TIF files. However, the TIF files were generated with a proprietary header that prevented them from being opened by any other software. That would have been nice to know at the onset, but it was an innocent oversight.
We began to consider alternatives for removing the headers. One method would be to open the TIF files in an Active Server Component, strip off the header, store the image locally, and display it through the Java applet. This would be difficult and time-consuming, but it was a possibility. We also found that the document management system had an API that included an export function, and we could use that function to strip off the headers.
We decided to go with the API since trying to strip off the header manually would have been impossible, since we later determined that the optical drive wasn’t really accessible through windows explorer. It had a very efficient disk format, but it certainly wasn’t FAT or NTFS. So the only way we could get to the images was through the API.
We reviewed the API documentation and talked to the company that developed the document management system. Now that we had a direction, we informed the client of what we needed to do and that this obstacle would require adding more time to the project. Once we got the okay, we started coding. Note that it is very important to keep your client, or your own company, informed of obstacles that may impact your time line and cost/budget.
We took our first version to the client location for testing, only to discover that it wouldn’t work because the document management system was a 16-bit application. Who would write such a system with a 16-bit compiler and not update it as time progressed? We were told that the original system was written for Windows 3.1 and that a 32-bit version would be released soon.
Anyway, we had to move on. So we dug up an older C++ compiler and rewrote our program. The system was finally complete. We could step back and look at our accomplishment, the best part of any project.
How this section relates to you:
When you run into a problem during development, take a step back and analyze the issue from a larger perspective. Ask yourself the following questions: How big of a problem is it? Are there obvious or simple work arounds? If there is a simple method of solving the problem, go with it. If the problem is too complex, try breaking it down into smaller problems that can be dealt with more easily. I can’t remember a time that I have run into a problem that would be considered a show-stopper; there is always an alternative if you are willing to go the distance. Use all of the resources you can find. In most cases, someone else has seen the problem before and may be able to offer some suggestions or insight that may help.
When asking a client what their deadline is for the system to be up and running, the typical response is "Yesterday!" Everyone wants to get the system developed quickly, but it is important that quality and scalability be among your highest priorities. Above all, be realistic with your time line. You have to consider the complexity and your resources.
It is important to build in time for the potential issues that you may encounter along the way. If you get done early, the client will be delighted. But you must manage your client’s expectations from the start and keep them abreast of any situations that may arise.
Never Surrender
Web-based systems integration is a real opportunity for challenge and growth, and a way to decrease the number of competitors you run up against. Furthermore, the additional skills and experience make you more marketable.
I would like to hear about some of your projects, the challenges you face, and how you overcome the obstacles. My email address is mailto:ttanner@activelogic.com.
Developers who don't want to spend a lot of money on SQL Server and who want a database that's more robust than Access may find MySQL to be a pleasant alternative. This introductory article covers the bare essentials for getting MySQL installed and running in the Win32 environment. [Read This Article][Top]
There is never a guarantee of project success when endeavoring to build a sophisticated application. However, there are established steps to follow that will ensure a clear, concise scope, support for the team involved, and a solid opportunity for successful deployment. [Read This Article][Top]
If your SQL Server is exposed to the Internet, then hackers are probing it. This article shows how to secure a SQL Server database that's being used with a Web application [Read This Article][Top]
Feel intimidated by .NET? This article by Rob Chartier is designed to ease any level VBScripter (ASP) into .NET by clarifying some OOP concepts. [Read This Article][Top]
A few members of the 15 Seconds discussion list talk about the proper way to use methods in order to prevent ADO object errors. [Read This Article][Top]
Solomon Shaffer explores the life cycle of an ASP.NET page from initialization to unloading. He also explains the various methods to override ASP.NET server-side events. [Read This Article][Top]
Rob Chartier creates a simple portable and reusable address book in .NET to demonstrate the power of N-tier application architecture. Complete source code included! [Read This Article][Top]
Learn about N-tier application architecture and realize that developing with multiple layers produces a flexible and reusable application for distribution to any number of client interfaces. [Read This Article][Top]
Learn about N-tier application architecture and realize that developing with multiple layers produces a flexible and reusable application for distribution to any number of client interfaces. [Read This Article][Top]
Mailing List Want to receive email when the next article is published? Just Click Here to sign up.
