Scalability - How to keep data flowing as registration volume increases

I spent two drizzly days in Seattle last week visiting some of our best clients, including Weyerhaeuser and Safeco. With our phenomenal growth over the past two years (we've processed over 2.7 million registrations in 2006 - already twice as much as last year), we continually have to upgrade our servers and network equipment in order to handle the increased load. So what can we do to make sure that registration forms respond quickly when 10,000 people try to log onto the system at the same time?

The first thing to realize is that event registration is a very demanding web-based process. A single page of an online registration form will require 10-100 times as much computer processing power as a single page of a content-driven web site such as Yahoo! This is because registration systems involve a great deal of data storage and retrieval, while news sites simply deliver static web pages to viewers.

So to keep the registrations flowing, we have to eliminate bottle-necks at every step of the process:

1. Registrant requests a page from their browser
2. Certain's servers receive the request and pass it to one of several web servers
3. The web servers handle the request and retrieve and store data into the database
4. The database server processes individual queries to read or write information
5. Backup systems make copies of the data to disc and magnetic tape
6. Security systems monitor the process to prevent unauthorized intrusion

All of this processing must occur within about a half-second, or the registrant will "feel" like the system is sluggish.

So to go from a million registrations per year to two million, to four million, to ten million, we continue to make large investments in our network operations.

Network

The first requirement is to make sure that data can get in and out of our servers fast enough to handle all incoming page requests. Internet "bandwidth" is a measurement of how much data can be transmitted in a given period of time. Internet Service Providers (ISP) measure bandwidth by both average usage and peak usage. For example, in September 2006 we increased our bandwidth from "1/10" to "2/100". This means that we pay for a monthly average usage of 2 Mbps (Megabits per second), but we are allowed to use up to 100 Mbps during peak periods.

For comparision, an average web page has about 100 Kb (kilobytes) of text and images. That means that the web server had to deliver 100,000 bytes, or about 800,000 bits since there are 8 bits in a byte. (Note that these calculations aren't exact, due to various small descrepancies between measuring systems of disc, computer, and network engineers.) With a million bits in a megabit, you would need 0.8 Mbps to receive the page in a second. High definition digital cable requires about 20 Mbps to deliver HD movies to your monitor.

Our current daily average bandwidth usage is just over 1 mbps, and our peak usage is around 15 mbps, so we have plenty of room for future growth and spike loads.

Software Foundation

Another way to improve performance is to upgrade the underlying software that the web and database servers use to deliver the page requests. We currently are upgrading our database servers from 32-bit Microsoft SQL Server 2000 to 64-bit Microsoft SQL Server 2005. This allows the database to process requests for information much faster. In addition, SQL Server 2005 allows for simple configuration of clusters of multiple servers. Instead of using one database server for processing and another as a backup in case of failure, we can use the processing power of multiple servers, which improve performance during normal operation and provide for redundancy during periods of failure.

Data Storage

With a data intensive application such as online registration, the overall performance can be limited by how fast the servers can read and write the data to disk. We are implementing high-speed Storage Area Networks (SAN), which improve speeds by orders of magnitude for more rapid read/write actions to discs and for database transaction log backups.

Application Architecture

Once the network, hardware, and software foundations are ready for high performance, it's up to the application to deliver.

• First, we are modifying Register123 to “cache” information, such as event and form configuration, that does not frequently change. Data stored in the cache is located in the ultra-fast memory of web servers, so when a page request comes in, the information can be delivered immediately instead of requiring a round-trip request to the (relatively) slower database servers. The specifics of caching are complex, but basically if the information has changed, then the web servers will automtically update the information in the cache from that in the database (this is called "refreshing the cache"). Caching will reduce database utilization by up to 35% per page without any sacrifice to the Users' experience.
• Second, we are going to implement a queue system for incoming requests. During the vast majority of time, registrations will be handled immediately when our servers receive the request. But in rare periods of exceptionally high demand (when database utilization exceeds 75%), incoming requests will be placed in a queue and handled on a “first-come, first-served” basis. While in the queue, registrants will see a “System Processing” message, much like the ones on online travel sites that so many Internet users have seen before. Their total wait time will be a few seconds (depending on system load), which is vastly superior to not being able to access the system at all.

Performance improvement is a never-ending battle with increasing registration volume. It is one that Certain Software is committed to win.

The future of meetings technology companies

Last week, Don Lipper of WRITE THE FIRST TIME asked me some questions for an article he is writing for Perspective Magazine about the future of meeting planning technology.

Q. Technology will only go so far - what services require a human touch? Will that always be the case?

I anticipate that the following services will continue to require a human touch:

• Creating new business relationships that involve the exchange of large sums of money.

This will always require a human touch; however, the definition of "large sums of money" will continue to increase. For example, in 1995, most people would not buy a book for $50 online because of fears about the security and integrity of the transaction. By 2000, many people were comfortable buying a $500 airline ticket online, without speaking to an agent. In 2005, some people are willing to buy a $25,000 car online at eBay Motors, without meeting the seller. Today, most people would not buy a $250,000 house without meeting someone (the owner or agent) in person, but in the future that may change. But it will be a long time before a business will engage in a $25 million transaction without at least once meeting the owner or agent, looking them in the eye, and shaking their hand.

• "Greeting" and "Host" positions

You may book a plane ticket online, drive yourself to the airport, ride an automatic train to the terminal, and check-in online; but if there isn’t a human standing behind the counter when you arrive, you’ll think the airline has folded overnight.

In the future, however, the positions where human greeters are expected will diminish. Future chains of budget hotels could run automatically, with self check-in / check-out and automated cleaning. But humans will always be needed to handle the situations that the software programmers did not anticipate.

• Having fun

"Experience-based" meetings work for networking because they are fun, and you remember people who you have fun with. Even the most die-hard video gamer who lives alone, works at home, has pizza delivered three times a day, and plays MegaQuest 2000 all night will likely have a network of similar recluses scattered around the world who play the team version of the game online and text chat for hours each day.

• Doing something new

Technology is great at doing the same thing over and over, and advanced technology can anticipate changes in its environment and adapt automatically. But humans are needed to do something truly new and creative. However, after the "new" process, event, or product has been done a few times by humans, watch out because the technology will take over and humans will have to find something else that is "new".

• Being inspirational

Think of the best motivational speakers you’ve heard at an event. A machine could run a laser sound and light show and chant slogans and display images of what you should do and what you should not do. But people want to see a human standing up there – someone who makes them think, "this person is just like me – and look what they did". That experience inspires behavior change and recharges the soul. Technology will never be able to do this because you can’t relate to technology – instead you would think, "machines have it easy because they don’t face the same obstacles that I do".

That said, technology can help people who are truly inspirational to spread their story faster and to a wider audience.

Q. Where do you see the most consolidation in the future?

• Group Travel from the attendee perspective: Registration, Housing, Travel.

It will be easier for attendees to confirm that they are going to a meeting, and then have their air, ground transportation, hotel, activities, meeting agenda, maps and FAQs completed and confirmed in a single itinerary

• Group Travel from the meeting planners perspective

The hotel, the air travel agency, the event planner, and the suppliers will each have their own system that best meets the needs of their business. But their systems will be able to "talk" to each other so that when the travel agent changes a flight from Monday to Tuesday, the Meeting Planner’s system gets the change, and automatically sends it to the Hotel to update the room reservation.

• Meeting management and attendee management

Information about the event – from the meeting request to the hotel RFPs to the Event Specification Guide – will integrate seamlessly with attendee data management systems. So when a registrant cancels their Wednesday night dinner in the online registration system, the BEO (Banquet Event Order) for the hotel will get the change, and the budget and expense management systems will be updated automatically.

• Small technology providers will merge to form larger companies

As products and business models mature, the many innovative small companies that were created around event technology in the past 10 years will continue to come together to form a smaller number of larger, more efficient technology providers.

Q. What is driving that consolidation, is it acquisition of territory or technology?

The consolidation of companies is being driven by the maturation of the technology providers’ business models, the desire of early-stage investors to realize a return on their investment, and the market demand for more comprehensive (and thus more complex) applications that will address all of their concerns. Small companies have overlapping overhead expenses that can be eliminated through mergers. Combining customer bases provide increased revenue for greater investment in technology enhancements. And complex software, by its nature, is expensive to develop the first time but very inexpensive to repeat for multiple customers.

Today, it is so easy for technology companies to operate nationally and internationally that local territory strength is not as big of a driving factor for these mergers as are the financial and technology reasons.

Q. When the dust settles five to ten years from now, who will be the winners? Will a single standard platform emerge or will there always be a place for focused standalone best of breed applications alongside with total solution suites?

The big winners will be the meeting planners – unless they allow one company to dominate the marketplace. The meetings technology industry will be better-served by a few major providers then it is now by scores of small players. But competition is healthy for the market as it keeps costs reasonable for planners and drives technology companies to achieve their best.

My hope (and expectation) is that the single platform that emerges will be the set of data communication standards being developed by APEX (through the Convention Industry Council) and OTA (the Open Travel Alliance). Technology providers will build systems that fit their customer base niche, but adhere to the open standards when they communicate with other systems. I expect that each segment of the group travel market will have a few major players who offer a single platform, but that the systems will be different for the different segments. For example, meeting planners will have 2-3 vendors providing integrated meeting and attendee management software – this makes more sense for them then having one meeting management system and another attendee management system. But hotels will have completely different systems (maybe provided by a few different providers), travel agencies will have yet another suite of tools, etc. It makes little sense for a travel agency or meeting planner to have 2 tools in their organization, but it makes less sense for travel agencies and meeting planners to be using the same tool that does everything for everyone. A more practical model is for major technology providers to produce a single best-of-breed suite for each market, and then these suites communicate via open XML standards such as those defined by APEX and OTA. This is why I joined the APEX Technology Advisory Committee and spend hours each month contributing to the development of these standards.

Looking ahead even further, once the major platforms establish a large customer base and adopt the open standards, new opportunities will open for niche applications. These niche applications could use the open standards to extend the core functionality of the major platform in order to provide a specific feature needed only by one relatively small market niche. The major platforms will not be able to invest in development of highly-specific features for small market niches, but a small business could fulfill that need while integrating within the major platform so that the overall experience of the end user is seamless – from the meeting planners perspective they think they are using one application. For example, a provider of photo-ID name badges for events could use the open web services to add their functionality to the 2 or 3 major registration system platforms. Such a function fills too small of a niche for a mainstream registration product to address in its core platform, but it would fit in nicely as an add-on product so that the meeting planners who need it don't have to use a separate application to fill this need.

APEX and the future of meeting data transfer

Yesterday, I spoke at the Society of Government Meeting Professionals (SGMP) North Texas chapter luncheon at the DFW airport Hyatt Regency. The Hyatt just finished a renovation of there meeting space, and if you need a central location for a nation-wide meeting then this is a good place. DFW is within a three-hour flight of most of the U.S., and the hotel manager seems to be willing to make deals in order to fill this new space.

I created a new PowerPoint for this presentation, and I think it worked well for describing to a non-technical audience how APEX is going to help make the lives of average meeting planners better. I'll attach the slides after this post.

The future of data exchange in the meetings and travel industries: Open Travel Alliance

The Open Travel Alliance (OTA) is an association of companies involved in the global travel marketplace. Most major airlines, hotels, car rental and ground transportation companies, cruise lines, and activity operators participate to some extent in the OTA.

The primary goal of OTA is to "transform the travel industry into a single global marketplace of products and services... through development of a commonly accepted communications process using XML". This means that the OTA will develop a common data language, which participants in the travel industry will agree to use in their data communication with one another.

The OTA publishes its XML standards twice a year (June and December), and anyone can download the current standards at after completing a free registration.

Currently, the Convention Industry Council (CIC) represents the meetings (group travel) industry to the OTA through its Accepted Practices Exchange (APEX) initiative. Members of the APEX Technology Advisory Committee (TAC), like EJ Siwek and myself, work with the OTA's Hotel Working Group (HWG) in order to produce XML standards that meeting professionals can use to communicate with their Hotel trading partners.

APEX has made progress in this work with OTA.

• The Hotel RFP standard was included in OTA's 2005B release (December 2005)
• The Rooming List standard was included in OTA's 2006A release (June 2006)
• The Event Specification Guids (ESG) will be included in OTA's 2006B release (December 2006)

Extensible Markup Language (XML) provides a text-based means to give structure to information. For example, if I tried to give my contact information to a computer system as I would to a human, e.g.

Rick Borry
rborry@certain.com
415-353-5330

It is doubtful if all systems would correctly interpret my information. If I apply XML to the information, however, then any system that understands the XML standard could correctly interpret my information, e.g.

In this manner, my contact information has structure, and the computer systems that pass the data back and forth can clearly understand that "Rick" is my first name and "Borry" is my last name.