An end-to-end eCommerce solution requires more than a fancy website

Introduction

The front-end website

The back-end application

The shared database

Back-end processing is not simple

Integrating the front end with the back end

Is there a back-end application that I can integrate with my website?

Conclusion

References

Comments

Introduction

The demand for websites on the internet is constantly growing, and so is the army of developers who want to earn money by building these websites. When a company wants an eCommerce website for their business, a website which allows customers to purchase their products over the internet, they turn to this army of website builders and expect an end-to-end solution which satisfies their needs. Yet what they, and even the website builders themselves, fail to realise is that building a sexy-looking website just provides the front end. A proper end-to-end solution must also have a back end, and this back end operates under a different set of rules, is much bigger than you would first think, and requires a different set of skills from the developers.

The front-end website is the equivalent of the front office in an organisation as this is the principle point of contact with customers or clients.

The back-end application is the equivalent of the back office in an organisation. It is less visible to the outside world, but it is still a vital part of the business.

In order to provide a proper end-to-end solution it is a common mistake to assume that you can start with a front-end website and simply "extend" it to include the extra functionality required by the back end. The back end is not an extension to the website, it is a separate application in its own right. It should also be remembered that back office applications existed long before the advent of the internet when all the customer interaction took place in physical stores and not websites. In some cases the back end application is not a single application but a collection of smaller applications each dealing with a separate area, such as "order entry", "inventory" and "shipping". Sometimes this collection of back-end applications has little or no integration, which makes it difficult for users (members of staff) to view data which spans application boundaries.

The front-end website

The main purpose of the front-end website is to look good so as to attract as many visitors as possible. The actual functionality is always quite simple - show the customer what he can buy and encourage him/her to buy it. There will usually be the need to connect to a payment gateway to take the customer's payment, and to communicate with a back-end database, but this communication is usually quite limited - "read products", "add customers", "add orders".

In the very early days all the "purchase" button did on a website was to generate an email. A salesperson from the company would then contact you to arrange payment, either by sending a cheque through the post or by taking your credit card details over the telephone.

What skills does the front-end website require? As it is a web site it requires HTML, plus CSS for styling, Javascript/Ajax for fancy interactions, and lots of sexy graphics plus some Search Engine Optimisation (SEO). There is not much "real" programming involved.

People who call themselves "web developers" may only have a limited set of skills:

1. Some are nothing more than graphic designers who can produce pretty images. They can show you what they think the website should look like, and describe how it should function, but they cannot actually build anything that works. This means that somebody else has to translate these pretty pictures into working HTML documents and program code.
2. Some can take a design and build a mock-up of the website in HTML but with dummy data instead of real data from a database. They can also incorporate some CSS and perhaps some Javascript to enhance the look and feel of those web pages. Somebody else then has to write the program code which generates each of these HTML documents, and replace the dummy data with calls to the database.
3. Some developers can design the web pages, build the HTML, CSS and Javascript, and write the program code which joins those web pages to the database, but they still rely on a "back-end developer" to design the database(s), write the data access code, implement the business rules, and build the back-end application which maintains the contents of the database(s).
4. Some developers can actually handle both the front end and back end processing, but as each of these ends requires skills which are not relevant in the other, the number of these developers is quite small.

Some non-eCommerce websites make use of pre-written packages which deal with Content Management (CMS) or Customer Relationship Management (CRM) which can be either free or low cost. An amazing number of people seem to think that it is a trivial exercise to install one of these packages and extend it to provide both a front-end and back-end eCommerce solution, but attempting to bend a CMS or CRM package to do what it was not designed to do is guaranteed to end up as expensive, inefficient at best or a total failure at worst.

The back-end application

Anybody who knows business should be able to tell you that although the front office is more visible to the outside world, what happens in the back office is much more complicated, and the front office couldn't function without it. "Taking a customer's order" is just the first step in a series of events which ends with "product delivered to customer". This series of events is commonly known as Order Processing, Order Fulfilment, or even Enterprise Resource Planning (ERP).

Once the customer has entered his order via the front end these order details must be made available to members of staff in the back office so that they can process the order all the way to delivery to the customer. This processing may require several stages, such as "verify payment", "authorise order", "pick from inventory", and ending with "ship to customer". There may be additional stages such as "assemble product" or "manufacture product", or even "obtain product from supplier". It is important that this processing is performed as quickly and efficiently as possible as a dissatisfied customer will be bad for business.

It should be obvious at this stage that the front-end website and back-end application are two different entities.

• The front end can be accessed by anybody, whereas the back end can only be accessed by members of staff.
• Search Engine Optimisation may be important for the front end, but for the back end it is totally irrelevant.
• The front end has the ability to take orders while the back end requires the ability to process those orders, and this requires a larger number of different screens with different functionality.
• The front end is more concerned with visual effect and sexy graphics than functionality, whereas in the back end it is the data and the functions that can be performed on that data which takes precedence. Members of staff want a system that allows them to get their job done as quickly and efficiently as possible, so clumsy screens which hinder their work are no good at all however sexy they look. The screens in the back end are more concerned with displaying data which occurs in columns and rows, either a single row to a page or multiple rows, so it is easier for these screens to conform to standard patterns.
• The back-end application may contain many functions, and not all of the these functions may be available to every member of staff, so it will need the ability to restrict access to certain functions to authorised users. This ability is usually provided by a Role Based Access Control (RBAC) system which groups users into roles, then allows each role to access a particular set of functions.
• This also implies that every user will have to pass through some sort of login/authorisation process before they can access the system. This login process may require either Single Factor or Two-Factor Authentication (TFA) or Two Token Authentication (TTA), possibly using One Time Passwords (OTP).
• The back-end application need not be written in the same language as the front-end website.
• The back-end application may only be available as a desktop application and not over the internet or the organisation's intranet.

The fact that the front-end website and back-end application have such significant differences, especially when they are written in different languages, usually means that they have different teams of developers with different skills. Even if the same development language is used for both parts it does not automatically follow that a single developer will have the skills to cover both parts. A front-end specialist will still struggle with the functionality of the back end, just as a back-end specialist will struggle with the fancy graphics of the front end. Developers who can create sexy looking websites are ten a penny, but the number of developers who can design and build entire ERP systems is quite small by comparison.

What skills does the back-end application require? If it is built as a web application then HTML and CSS are required as standard, but the main requirement would be in-depth knowledge of the business processes required in the back-office application, and the ability to design and build the software to carry out those business processes. This comes in several parts:

• The ability, usually in conjunction with a business analyst, to analyse the business requirements in order to identify what data needs to be processed and how it is to be processed. Identifying the problem is the critical first step, and must be completed before you have any hope of identifying the possible solution.
• The ability to design and build the database(s) necessary to support all the business requirements. This requires the ability to perform data normalisation so that there is a place for every piece of data, and every piece of data is in the right place. A well-designed database is the bedrock on which a successful application can be built, whereas a badly designed database will always be a millstone around the necks of the application developers.
• Having a well-designed database is no good unless you have the correct software to process that data. This is not just a matter of designing a function which allows you to move data between the User Interface (UI) and the database, with or without transformations, it also requires the processing of business rules so that certain processing is only allowed when the circumstances are right. It is this identification of the business rules, and the ability to create the code to process those rules, and to make sure that the right code is executed at the right time, which separates the code monkey from the serious developer.
• Knowledge of SQL is also important as this is the standard language for managing data in a Relational Database Management System (RDBMS). Complex queries may be required to pull together data which could be spread across multiple tables, and there will usually be the requirement for management reports which pull together large volumes of data to present them in summary form. Writing a complex query which runs as efficiently as possible is sometimes beyond the capabilities of the average front-end programmer.

The shared database

If the purpose of the front-end website is to take orders while the purpose of the back-end application is to process those orders it should be obvious that these two entities must share the same data, which is usually held in a relational database (RDBMS). It would be rather inefficient for the front and back ends to have their own databases as it would require additional processes to transfer the data from one to the other. This would be even more complicated if they had totally different formats and structures. Having said that, it has been known for some companies to use a small MySQL database for the front-end website, then have processes in place to transfer that data to a larger Oracle database for use by their legacy back-end application. While this may be a practical solution for those who have a large financial investment in a legacy back-end application, it is far less attractive to a company which is starting from a clean slate.

You may think that the database only needs to have 3 simple tables - PRODUCTS, CUSTOMERS and ORDERS - but in the real world that idea wouldn't stand up for 5 minutes. In order to cater for the data requirements for a wide variety of organisations which sell different products under different circumstances the experienced developer will tell you that each of those areas has such a wide ranging number of possibilities that they require separate databases, not just separate tables. Let me explain what I mean:

The PRODUCT database may require the following tables:

• product data, such as id and description. Note that a product may have more than one means of identification, external as well as internal.
• product features, such as colour or size, which may be applied to any product.
• price components, with the ability to specify a change of price on a particular date, or different prices under different circumstances. This could actually become very complicated:
  • You may start with a base price for each product with the option of a separate contract price for selected customers, either for individual customers or classes of customer.
  • There may also be additional charges for certain features.
  • Surcharges such as delivery charges may have different amounts for different geographical areas.
  • Surcharges such as sales tax may have different percentages for different countries.
  • Some products may be taxed at different rates (standard, reduced, zero or exempt).
  • Discounts may be either fixed amounts or percentages and may be applied globally, seasonally, for individual customers or classes of customer, or by a discount voucher.
  • Discount vouchers may be single use, multiple use, or restricted to a range of dates. Discounts may also be of the type "buy quantity X and get quantity Y free" or "buy product X and get product Y free" or even "spend more than £X and get Y% off".
• product categories, to break down the collection of products into different groups. A product may belong to more than one category, and a category may be comprised of a hierarchy of other categories.
• product feature applicability, which identifies which features are actually valid for which products.
• product feature interaction, so you can identify which features are either compatible or incompatible with each other.
• product components, where a product may be comprised of a number of other products.
• compatible products, so you can tell your customer "if you like that you may also like this".

The CUSTOMER database may require the following tables:

• Customer details, such as id and name. Note that a customer may have more than one means of identification (such as a PayPal ID).
• Address details. Note that a customer may have more than one address (main address, delivery address, etc). It becomes more complicated if you want to share address components among many users
• Contact mechanisms, such as email address, telephone number, mobile number, fax number. Note that a customer may have multiple contact mechanisms.
• Customer classifications, allowing you to group customers into whatever classifications your organisation sees fit. Note that a customer may belong to any number of classes.

This can become complicated if a customer can be either a person (as in a business-to-consumer (B2C) website) or an organisation (as in a business-to-business (B2B) website). If the customer is an organisation then the system will need to know the name of the contact person at that organisation.

With a B2C website it is usual to take the customer's payment before the order is processed, but with a B2B website it is usual to take the order without payment, then send the customer an invoice with payment due within a certain time limit.

What happens if you also have to deal with suppliers? Do you have separate tables for customers and suppliers, or can you merge their details into a single table? What happens if some people/organisations can be both customers AND suppliers?

The ORDER database may require the following tables:

• order header, to hold order id, customer id, order totals, delivery address, and possibly (for international systems) currency code and exchange rate.
• order items, as there may be more than one item per order.
• order adjustments, such as discounts and surcharges, delivery charges or sales tax.
• order item adjustments, as these may apply to individual items instead of the whole order.
• order item features, to allow zero or more features for each order item.
• order terms, if there is the option for different terms to apply under different circumstances.

As you should be able to see, the actual data requirements cannot be met satisfactorily with just a small number of simple data structures. You have to have a database design that allows as many options as possible, and software which can deal with all of those options. You may not think that all of those options are actually necessary, but those of us who have been working in the real world for long enough know that an organisation's requirements have a habit of changing and becoming more complicated over a period of time, so it can be worthwhile to have a system which has a lot of the common options available from the start so they can be quickly turned on as and when required.

As the shared database must cater for the needs of the back-end application, which is much more complicated than the front-end website, it follows that the database should be designed to satisfy the needs of the back-end application, and the front-end website should be altered to deal with that design.

The back-end application may also need additional databases for additional functionality, such as "invoicing", "inventory" and "shipments".

Back-end processing is not simple

Building a website that looks sexy is one thing as it is all glitz with very little functionality that requires "real" programming. This is something that a lot of website designers can do with their imaging software and a bit of fancy CSS and Ajax (Javascript). Building a back-end application is totally different as the glitz is irrelevant and the functionality is everything. You have to be able to design and build a database to hold all the possible variations in data, then you have to design and write the code which manipulates that data in the way which is most effective for the business. You have to understand the business processes which are involved, then be able to translate all those business processing rules into effective code.

Creating a website which takes customer orders is quite simple, but can you write an application which progresses those orders all the way from order authorisation, picking from inventory and then shipping to the customer? If you have customer shipments then you must be able to deal with customer returns. If you have inventory then how is that inventory replenished? Do you raise purchase orders on suppliers, do you manufacture parts or assemble them? If you deal with purchase orders then you must also have a mechanism for dealing with purchase returns.

If you have inventory then you need to record what stock is stored in which container at which location, which may be a different room, a different building or even a different address. You will need to record all receipts into stock as well as issuances from stock, and allow for variances following a stock check. You may also need a means to reserve stock so that when an item is selected for one sales order it cannot be selected for another. You may also need mechanisms to move stock from one container to another within the same location, or between different locations by means of transfer orders (both in and out). You may also need a mechanism to warn you when the on-hand quantity of certain items goes below a certain level so that you can start the replenishment process.

Once you have taken a sales order there may also be a requirement to turn this into an invoice, convert this invoice into a PDF document and email to the customer. If you have invoices then you will also have to deal with credit notes.

Do you want the system to track all activities for each customer? Do you want the system to send out emails? Do you want the system to act as an email client and track all communication with each customer? If you do then you are venturing into the realm of Customer Relationship Management (CRM).

Integrating the front end with the back end

The front-end website and back-end application are separate entities, with the only common ground being the shared database. This is shown in Figure 1:

Figure 1 - Separation of front-end website and back-end application

In this example the front end and back end are entirely different applications, almost certainly developed by different teams of developers, with different skills, and more than likely written in different programming languages. If there is any business logic which is required in both parts, then the programming code to implement that logic will have to be duplicated as it cannot be shared.

This inability to share code between the front end and back end results in an expensive duplication of effort. It may also lead to errors if any changes to the business rules are not synchronised properly in the two applications. This situation can be alleviated somewhat if the two parts are written in the same language, and if the back-end application has been developed around the 3-Tier Architecture (see Figure 2)

Figure 2 - The 3-Tier Architecture

In some large organisations it has been known for them to have separate teams of developers to work on each of the different layers even though they are using the same programming language. This is because the presentation layer is concerned with HTML, CSS and Javascript, the data access layer is concerned with SQL (and possibly database triggers and stored procedures) while the business layer is concerned with enforcing the business rules, data validation, flow control, etc, and does not deal with any HTML, CSS, Javascript or SQL.

With this architecture any one of the layers can (in theory) be modified, or even replaced, without having to make corresponding changes to any of the other layers. This means that the front-end website can have its own presentation layer (User Interface or UI) and share the same business layer and data access layer components that the back-end application uses. The business layer does not know (or even care) which presentation layer is communicating with it - it simply receives requests and returns the responses, and it is up to the presentation layer to deal with each response as it sees fit. This is shown in Figure 3:

Figure 3 - Different Presentation layers sharing the same Business & Data Access layers

In this scenario the code within the front-end website is concentrated around the fancy user interface, and when it wants to communicate with the database it accesses one of the many business objects in the back-end application with a simple method call, such as $object->getData() or $object->insertRecord(). This means that the front end is as light as possible, with all the heavy lifting, data validation, business rule processing and SQL query construction handled entirely by the back end business layer components. As all the business rules are handled in a single place, the business layer components, then changes to business logic are made in a single place and are automatically made available to both presentation layers.

For this to work all the application code must exist on the same server.

If the back-end application has been enabled for web services (ie. it has a component which accepts web service requests) then would be possible to change the front-end website to include a web service client in order to communicate with the back-end web service server. This is shown in Figure 4:

Figure 4 - Linking the front and back ends with Web Services

This method has two distinct advantages:

• The front-end web service client communicates with the back-end web service server by issuing an HTTP request over the internet. The server receives the request, processes it, and returns its response back to the client over the internet. The two servers need not be located in the same place, in fact they can be on opposite sides of the planet.
• Because both the HTTP request and HTTP response are language-agnostic it does not matter if the front-end website and back-end application are written in different programming languages. All that is required is that each language be able to deal with web services and XML documents.

Is there a back-end application that I can integrate with my website?

All this talk about integrating a front-end website with a back-end application is all well and good, but does such an application exist? Well, now you come to mention it, yes it does. In 2007 I built my TRANSIX ERP application which has the following characteristics:

• It is written in PHP, one of the most popular languages for writing websites and web applications.
• Because it is written as a web application it does not need to be installed on individual PCs, and can be accessed over the internet or the organisation's intranet.
• Because it has been built using the RADICORE framework it automatically includes all the RADICORE security features.
• It was designed around the database designs found in Len Silverston's Data Model Resource Book. These are flexible yet powerful designs which have proven themselves in many different industrial sectors. TRANSIX has implementations of the following databases:
  • PARTY - for organisations and people, customers and suppliers.
  • PRODUCT - for goods and services, features, categories and price components.
  • ORDER - for sales orders, purchase orders, transfer orders, order items, order adjustments, item adjustments, item features, non-conformance reports.
  • INVOICE - for invoices and credit notes.
  • INVENTORY - for inventory items, issuances, receipts, stock checks and variances.
  • SHIPMENT - for customer shipments and returns, supplier shipments and returns, transfers in and transfers out.
  • WORK-EFFORT - for time sheets and expenses.
  • REQUEST - for requests, requirements and quotations.
  Although each of these exists as a separate subsystem within TRANSIX, they are all completely integrated, so it is easy to move from one subsystem to another, or to see data from different subsystems in a single screen or report. The TRANSIX application currently has over 200 database tables, 350 relationships and 1750 transactions (screens, reports and functions).
• It was generated using the RADICORE rapid application development toolkit which was specifically designed to use the database structure to drive the generation of working software components. It uses a purpose-built Data Dictionary to do the following:
  • Import the table details from the database schema.
  • Export the table class files and structure files to the application.
  • Generate working transactions by linking database tables with transaction patterns.

The TRANSIX application was also built around the 3-Tier Architecture, so it offers the benefits shown in Figure 3.

The TRANSIX application also includes a collection of web services, both client and server, so it offers the benefits shown in Figure 4.

If TRANSIX has been built as a package, what are the possibilities for customisation when a client has non-standard processing requirements? The core code is sacrosanct and remains the same for everybody, but custom variations can be introduced by means of plug-ins. These are scripts which can be dropped into any of the separate plug-in directories, with separate directories for each client. With these plug-ins it is possible to change a screen layout, a PDF report layout, or even the code which is executed for certain events. Changes to custom code may be implemented either in conjunction with or instead of the corresponding core code.

Conclusion

I hope this article has shown you that building an end-to-end eCommerce solution does not stop with a fancy looking website. The front-end website is just the tip of the iceberg with all the heavy lifting done by the back-end application. Think of the front-end website as the showroom in a high street store. It is designed to attract customers and make sales, but regardless of the activity in the showroom you should realise that all the important work takes place behind that little door at the back labelled "staff only". It is the work done by the support staff in the back office that makes the activity in the showroom look effortless, and without it the glitz of the showroom counts for nothing.

The front end website and the back end application are totally different entities with different goals. In some organisations they are built in different languages, which automatically implies that they will have two separate development teams. Even if the two parts are written using the same language it would be naive to automatically assume that they can be built by the same developer(s). They are still different types of application with different goals. Building a sexy front end and a functional back end require totally different sets of skills, and it is extremely rare to find a single individual who has the necessary skills to cover both areas.

The most workable solution for organisations who are venturing into eCommerce and who do not have a legacy back-end application to support would be to find a back-end application that can easily be integrated with a front-end website. When the website is constructed, usually with dummy data to start with, it can easily be modified to communicate with the TRANSIX database. This means that as soon as an order is created in the website it instantly appears in the back-office application and can be processed to completion. This procedure has been employed for organisations in fields as far apart as jewellery, baby clothes and beauty products, so it is a proven product with a proven track record and not just vapourware.

But what if an organisation has an existing website and small but inefficient back-end application? This is slightly more involved, but still possible. It just requires two steps instead of one:

1. Transfer existing data from the legacy database to the TRANSIX database(s).
2. Convert the website functions to access the TRANSIX database instead of the legacy database.

This later procedure, the replacement of a legacy back-end application, has actually been done with TRANSIX, so it is not unknown territory.

If you would like more details of the TRANSIX package you can contact me using this link or this link.

References

• Web Site vs Web Application
• RADICORE - Rapid Application Development Toolkit
• Using Radicore components in a front-end website
• Why you should build your web application back-to-front