Working with a stateless protocol

Introduction

The Differences

- Statefull

- Stateless

Maintaining State manually

Radicore session management

Other points to consider

- The GET method is not used for updates

- No primary keys are visible in any URLs

- Each script is self-executing

- Each script deals with one visible page

- All access must be authorised

- Transaction navigation is controlled

- Jumping to child screens is via buttons, not hyperlinks

- Back Button Blues eliminated

- Multiple sessions on the same client are allowed

Amendment History

Comments

Introduction

Before switching to web development and the stateless HTTP protocol I spent several decades developing client-server systems which were statefull. These came in several flavours:

• "Thin" client with dumb "green screen" terminals connected to a central mini-computer acting as both the application server and database server. Two different protocols were used:
  • Character mode - where each key press is transmitted to the server, then echoed back to the client.
  • Block mode - where the user fills in a form and presses the ENTER key to transmit the whole form to the server.
• "Thick" client with networked PCs connected to a central database server, with the PCs using a GUI.

You should notice that the old block mode protocol with its ENTER key is quite similar to the new HTML form with its SUBMIT button.

The Differences

Statefull

This requires the user to create an instance of the application on the client device, which then connects to the central server where it creates an associated instance. Both these instances, on the client and the server, are allocated a portion of memory which may expand or contract. Both instances remain active until terminated by the user, which means that any resources (e.g. memory) which has been allocated to an instance are treated as "locked" and cannot be used by other instances. This usually means that the number of concurrent users is limited by the total number of resources which are available on the server.

During the processing of an application the user may perform many actions, such as navigating through menus, selecting transactions, jumping from one transaction to another, et cetera. Because the application instance does not die until the application is terminated, anything which is stored in memory during one action is instantly available to any subsequent actions. In other words the condition or "state" that the application was in at the end of one action is automatically carried forward into the next action.

It is also possible for the application to send information to the client at any time, without that information being requested.

Stateless

This does not require the user to start an instance of the application as all access to the application is via a web browser. The application, which is running under a web server, knows nothing about any particular client until it receives an HTTP request. When it receives a request the following happens:

1. The web server creates and activates a child process to deal with the request, or it may reactivate an existing process which is currently "sleeping".
2. The child process deals with the request and generates a response, which the web server will send back to the client device.
3. The child process then dies, or puts itself in a "sleep" condition.

The idea behind a child process going to sleep instead of dying is to avoid the overhead of creating and activating a brand new process each time. With this method a process is created just once, and while it is not actively dealing with a request it sits in a queue of sleeping processes which are waiting to be reactivated.

It should be noted here that even if a child process does not die but merely goes to sleep it loses all memory of the request it just processed and the response which it generated. Even if the next request from the same client is given to the same child process there is no memory of any state left by the previous request. This is the stateless nature of the HTTP protocol.

• The HTTP protocol does not allow a process on the web server to be permanently allocated to a particular client.
• After dealing with a request from one client a child process may deal with many requests from many other clients before it receives the next request from the original client.
• When the web server receives a request from a client there is no way to guarantee that it will allocate the same child process that dealt with the previous request from that client.
• If that website is operating a server farm there is no way to guarantee that a subsequent request from a client will be allocated to the same server that dealt with any previous request.

This means that a client is only consuming resources on the web server during the processing of a request-response cycle. When a client receives a response there is usually a time delay before the next request is issued (if any), and during this time the web server can deal with many other requests from many other clients.

The advantage of this approach is that although there is a limit on the number of child processes which can be active at any one time, the number of active clients can be considerably greater due to the fact that the requests are usually staggered instead of transmitted all at the same instant.

The disadvantage of this approach is that the web server does not maintain any memory of the activities of any particular client, so each request is treated as a new request and not as a follow-up to a previous request.

There is no state maintained at the server end, but what about at the client end? It is true that the browser maintains a history, but this is nothing more than a history of requests that have been sent. If a request is re-transmitted it is treated as if it were a new request. The responses may be cached, but while stepping through the browser's cache there is no communication with the server, so what is being viewed is from the browser's memory on the client, not the server's memory.

Maintaining State manually

As you can see, with a traditional client-server application state is maintained automatically. It does not require any manual intervention on the part of the programmer. This is not the case with a web application, but what manual steps are available to maintain state between requests?

• Using arguments in the URL. This can lead to large and unmanageable URLs, and there is a limit to the maximum size of a URL. It also means that a user can edit a URL and effectively change state, with unpredictable results.
• Using hidden fields in a form. This can lead to large forms which are therefore slower to transmit. Even though the data may be hidden in the rendered document it is still possible for the user to see it by using the browser's "view source" option. It is even possible for the user to edit the source and change some of the values before they are transmitted back to the server.
• Using cookies. There is a limit to the amount of data which can be stored in a cookie, there is a limit to the number of cookies which can be active at any one time, and larger cookies will take longer to transit. Even worse, the client has the option to disable cookies altogether
• Using files on the server. In PHP this ability is provided through a facility called sessions which allows data to be recorded either in a disk file or in a database table. The advantage here is that all data is maintained on the server, so nothing is transmitted to the client (except for a session name and id). It is therefore not possible for the client to view or edit this session data.

Radicore session management

All sessions are handled automatically within the Radicore framework and require no additional programmer intervention. Session data is stored within the database, not the file system, which provides additional security on the session data as well as allowing the application to be deployed across multiple servers in what is known as a server farm.

A session begins when a user logs on and terminates either when that user logs off or when that session data is erased from the server, such as when there has been no activity within that session for a predetermined time limit. It is also possible for a client to have multiple sessions open at the same time and for each session to be independent of the other. Session data is loaded into memory at the start of each script so that the saved values are readily available to the script. When the script terminates the session data is written back to the database to save any changes.

Data for the current script is stored within the $_SESSION array and is only deleted when the framework decides that data is no longer required. Each time a script executes the framework will examine the session array looking for saved data for the script. If it exists then the script will automatically resume from where it left off, otherwise it will start from an initial state.

How does the framework decide that a script's session data is no longer required? This is determined by the navigation direction when the user moves from one script to another. There are only two directions for navigation - forwards or backwards.

• Forwards is when the user presses a navigation button or a popup button within a form to pass control to another form. The current form, now known as the "parent", is suspended while the selected form, now known as the "child", is active. It is also possible to select another form when within the child in order to create a chain of forms, such a A=>B=>C=>D=>E, etc. This "chain" will appear in the breadcrumbs area of the menu bar. Only the last form in the chain can be active, all previous forms are suspended.
• Backwards is when the user presses a button in the action bar, such as "close", "cancel", "choose" or "submit", to finish the current action, or uses the browser's BACK button. Control will then be passed back to the previous script whose identity is determined by the framework. Using the saved session data the previous (parent) script will be able to resume from where it left off, which involves re-reading the database to pick up any changes made by the child script. At this point the child's data in the session array will be erased.

By default when a script no longer appears in the current chain of suspended/active scripts its session data will be erased. For scripts of type LIST this behaviour can be altered by using the Keep Data on Exit flag in the Update Task screen. This will enable the current selection criteria, sort order and page number to be saved so that they can be reinstated whenever that script is reactivated.

More information on this topic can be found in Appendix G: Maintaining state with sessions.

Other points to consider

During the building of the RADICORE framework I encountered situations which could be dealt with in several different ways. These situations, and my response to these situations, are documented below.

The GET method is not used for updates

There are two basic ways for a client to submit a request to a server - the GET method and the POST method. The GET method has all its data on display in the URL, so it is visible to the user and easily editable. With the POST method the data is not visible to the user, therefore difficult to edit. Also, requests via the GET method can be bookmarked so that they can be resubmitted at any time. POST requests cannot be bookmarked.

A GET method can be included anywhere in a web page as a hyperlink, but the POST method can only be used with a SUBMIT button.

Although it is possible for the same request details to be sent by either method, as a general rule the GET method should be read-only and all updates should be performed via the POST method. Not following this simple rule could lead to problems - when a search engine examines your site it attempts to follow any hyperlink, and if this request performs an update then an update will be performed. POST requests require a SUBMIT button, and these cannot be selected by any search engine.

In the RADICORE framework all GET methods are read-only, and updates are only performed via the POST method.

No primary keys are visible in any URLs

I have seen some systems which include the primary keys from their database tables in their URLs, which means that a user could edit the URL and change the primary key to another value. This may possibly allow a user to see a record which is supposed to be "off limits", so it presents a security risk.

In the RADICORE framework no primary keys are exposed in any URLs. The identity of the current selection is maintained in the session data on the server, so this potential security risk does not exist.

Each script is self-executing

A typical user transaction has two passes - a GET request which assembles the current details and sends them to the user, and a POST request which accepts any changes from the user and than applies those changes (after successful validation, of course) to the database. I have seen some applications where the GET request is processed by one script and the associated POST request by another. This whole idea seems counter-intuitive to me as it is impossible to engineer in a traditional client-server application, and I see no advantage of doing it in a web application.

In the RADICORE framework each script for a user transaction deals with both the GET and POST methods. The advantage I see is that each script has its own area in the $_SESSION array, so when the POST method on a script is executed it is easy to locate the data that was available for the previous GET method on the same script.

Each script deals with one visible page

In some client-server systems I worked on in the distant past it was standard practice for a single program unit to deal with several related screens. I have seen this practice carried forward into some web applications where a single script can deal with several different HTML documents. My preferred choice is to limit each program unit to a single screen. My reasons are documented in Component Design - Large and Complex vs. Small and Simple.

In the RADICORE framework each visible page has its own script, and each script deals with no more than one visible page.

All access must be authorised

The RADICORE framework has been designed for administrative web applications which can only be accessed by authorised users. This means that nobody is allowed to process a user transaction until their authority has been confirmed. This authorisation process has two steps:

1. The user must be authorised. Each authorised user has his own user id and password recorded on the USER table in the MENU database, and this information is verified when passing through the LOGON screen. When a user successfully passes through the LOGON screen that fact is recorded in the $_SESSION array. Whenever any script is requested part of its initialisation procedure is to check the $_SESSION array for this authorisation. If this authorisation is not there the current script is aborted and the user is kicked back to the LOGON screen.
2. The user's access to each transaction must be authorised. Although a user may be authorised to access an application, he may not be authorised to access every transaction which is available within that application. Each user belongs to one user group or ROLE, and each ROLE is given a list of available TASKS that users of that ROLE are allowed to access.

In the RADICORE framework all the security details are maintained in and controlled by a Role Based Access Control (RBAC) system. When constructing the details for menu buttons or navigation buttons, any transactions (tasks) to which the current user does not have access are automatically eliminated from the display. This means that the user is only ever presented with options that he can actually access - if he can't access it then he won't see it, and if he can't see it he cannot select it.

Even if a user tries to edit the HTML source to access an unauthorised transaction, the framework will detect it and reject it.

Transaction navigation is controlled

In a lot of web applications the user is able to move from one transaction to another by means of hyperlinks. These use the GET method and have the current context, such as the identity of the record to be processed, built into the URL. This means that at any time the user can use a bookmarked URL, or edit the current URL, and jump to any transaction within the application.

In the RADICORE framework the ability to jump to other transactions in the application is via one of the following methods:

• Menu buttons - these are hyperlinks and use the GET method. The only options which are available here are menu transactions and transactions which do not require context. If a selected option is another menu then the current set of menu buttons will be replaced by another set. If the selected option is a user transaction then the script for that transaction will be activated and the bottom portion of the screen will alter to suit the needs of the selected transaction.
• Navigation buttons - these are SUBMIT buttons and use the POST method. There is a separate set of these buttons for each transaction, and they define the set of child transactions which are valid for the current transaction. Pressing one of these buttons may cause context to be passed from the current transaction to the selected child transaction. While the child transaction is being processed the parent will be suspended. When the child transaction terminates the parent will be re-activated.
• Popup buttons - these activate a special type of user transaction known as a picklist. Their objective is to allow the user to choose an item from another part of the system and have its details recorded in the field containing the popup button.

When I say "context" I mean the identity of the current database record, or the identities of several database records which may have been selected. LIST1 screens do not require context as they can retrieve any number of records. UPDATE1 screens on the other hand cannot function until they are supplied with the identity of a record on which they are to perform that function.

It does not matter whether the GET method or POST method is used, the request will be sent to the current script where it will be processed as follows:

• Check that the selected child transaction exists. This also converts the task_id into the a script_id.
• Check that the user has access to the child transaction.
• Create a new area in the $_SESSION array for the child transaction and fill it with details that it may require.
• Pass control to the child transaction.
• When the child transaction is activated it looks in the $_SESSION array for its set of data, and operates on that data.
• When the child transaction terminates it may have data that it needs to pass back to its parent. This is done by putting that data in that part of the $_SESSION array that is allocated to the parent.
• When the child transaction terminates it will delete the data in the $_SESSION array that it has just used.
• When the parent transaction is reactivated it looks in the $_SESSION array for any data which has been passed back to it, and takes the appropriate action.

This communication between parent and child transactions is particularly significant. It is up to the parent transaction to create the data in the $_SESSION array that will be used by the child. If the child transaction cannot find this data when it is activated it will immediately abort and jump back to the last valid transaction. If it can find data to be processed this data will be deleted when the transaction terminates so that it cannot be processed more than once.

Jumping to child screens is via buttons, not hyperlinks

In a lot of web applications where one screen shows several rows of data, one database record per line, and a method is required to jump to a child screen, the most common method is to make each field on each line into a hyperlink. If any of these hyperlinks is selected they will activate the same child screen for the current row on the parent screen. This method has the following disadvantages:

• The URL in the hyperlink can only point to a single child screen, and this has to be hard-coded in the parent screen (i.e. it is static, not dynamic).
• In order to identify which database record needs to be processed its primary key is usually built into the URL, and exposing primary keys is a potential security risk.
• The hyperlink can only activate the child screen for a single selected row in the parent screen. This means that if you wish to process another row you have to return to the parent screen in order to make another selection.
• The hyperlink usually jumps to an enquiry screen, so options to jump to an update or delete screen are shown as additional hyperlinks elsewhere in the line. This means that the "update" and "delete" hyperlinks are duplicated on each line.

In the RADICORE framework there are no hyperlinks for jumping to child screens, only SUBMIT buttons. This has the following advantages:

• Each child screen is represented by a single button in a single navigation bar, thus there are no duplicate buttons on each row.
• The number of navigation buttons on each parent screen is taken from the MENU database at runtime, so they are dynamic, not static.
• The identity of the record(s) to be processed by the child screen is not built into any button. Instead there is a checkbox against each row which should be checked ON to select that record, or checked OFF to unselect that record. This mechanism allows multiple records to be selected and passed to the child screen for processing
• Although the child screen may only be able to process one selected record at a time, if multiple selections were made in the parent screen then the child screen will include a scrolling area which will allow the user to move backwards and forwards through the selected records without having to return to the parent screen and make another selection.
• As this uses the POST method, not the GET method, there are no primary keys in any URLs, thus eliminating a potential security risk.

Back Button Blues eliminated

One of the problems with a web application is that it cannot control what requests are sent from the client's browser. The web browser is totally under the control of the user, and requests may be generated in any of the following ways:

• By pressing a hyperlink or a button in the current page.
• By manually typing in a URL.
• By editing the current URL.
• By selecting a URL which has been bookmarked.
• By picking out a URL from the browser's history, usually with the BACK button.

Apart from the first option it is possible that any of these requests could actually be invalid, such as attempting to repeat a request that should not be repeated. A database update which is inadvertently repeated may actually cause data corruptions from which it may be difficult to recover.

The most typical scenario is where the user visits a screen, performs an update, and is then taken to a second screen. The user then presses the browser's BACK button which causes the browser to re-issue the previous request, the one which updated the database. If you visit any web development newsgroups you will see a common question is "How do I disable the browser's BACK button?" The answer is always the same - "You can't!".

There is no way to prevent the client from ending an invalid request. All you can do is check each request for validity and take appropriate action if it is not.

In RADICORE the framework keeps track of all page requests in a variable called $page_stack. The current transaction is always in this stack. If a child transaction is activated then it is added to the stack after its parent, and its parent is treated as "suspended". If a transaction is terminated it is removed from the stack and its parent is reactivated. When a request is received it is compared with the current stack and checked for validity. This can lead to the following possibilities:

• The requested transaction does not exist anywhere in the current stack. In this case the request is rejected and replaced with a GET request on the last entry in the stack.
• The requested transaction exists as the last entry in the current stack. In this case it is processed as requested.
• The requested transaction exists in the current stack, but not at the end. In this case all following entries in the stack are removed before the request is processed.

This is documented further in Back Button Blues.

Multiple sessions on the same client are allowed

Another problem which can cause unexpected results is the fact that the user may open up more than one browser window to access a particular web application. The following methods are possible:

1. The user may create a new instance of the web browser and type in a URL for the same web application.
2. The user may clone a window in the current browser instance. In Internet Explorer this can be done by typing Ctrl+N.

The problem here is that the different browser windows will all share the same set of cookies. As the identity of the session on the server is by default held in a cookie on the client, by sharing the same session cookie the different browser windows will in effect share the same session data on the server. This may cause problems if the activity in one of the browser windows causes session data which is being used by another browser window to be modified or even deleted. By switching to another browser window the user may assume that the activities in that window can continue from where they left off, but if the session data held on the server has been modified then the results can be completely unexpected.

It is not possible to prevent the user from opening up multiple browser windows into the same web application, so the RADICORE solution is to give the user the opportunity to create a new session on the server for each browser window. The mechanism is as follows:

• The cookie holds a value in the format session_name = session_id where:
  • session_name is a name which defaults to PHPSESSID.
  • session_id is a 40-character unique string generated on the server.
• The only way to have multiple session_ids in force at any one time is to use multiple session names.
• The RADICORE framework uses session names in the format menuNN where NN is a number in the range 1-99.
• Just above the menu bar on every screen is an option called new session which, when pressed, has the following effect:
  • A new session name is generated by incrementing the number at the end of the current session name. A lookup is performed in $_COOKIE to ensure that the name is unique on the current client.
  • session_regenerate_id is called to generate a new 40-character session id on the server.
  • At this point the $_SESSION array still contains a copy of the original session data, so when the new session request terminates by sending the current page it has the following effect:
    • This copy of the session data is stored under its new session_id, which means that it is now completely separate from the original data with its original session_id.
    • The new session name, with its new session_id, is transmitted back to the client in the URL.
    • Each browser window uses a different session name and therefore a different session_id and therefore a different set of session data on the server.

This is documented further in Client Clones and Server Sessions.

Amendment history: