Dr. Robert Chandler | International Crisis Communication Expert
  • HOME |
  • CRITICAL THINKING |
  • ABOUT |
  • Contact

Improving Productivity, Participation and Satisfaction in Business Meetings

1/27/2017

0 Comments

 
Picture
In my years of teaching organizational and leadership communication one recurring topic centers on the challenge of conducting effective business meetings. Meetings are one of the commonly dreaded chores for many employees and far too many meetings deserve the disdain. Poor meeting performance can negatively impact productivity, derail decision-making and forward moves, damage employee morale and become a festering problem that drains attention, resources and valuable work time with only negative outcomes to show for the investment.

Over the decades, I have read/reviewed numerous academic and scholarly studies about meetings as well as read a long list of popular professional viewpoints on steps to improve meeting performance. While the body of literature is too large to attempt to summarize in these few short paragraphs, I thought it might be helpful (to someone) to hit a few of the more frequently mentioned “tips” which have been recommend for improving productivity, participation and participant satisfaction in business meetings. I am confident that I picked these ideas up from many others over the years, so I defer credit to numerous unknown sources for the following 10 key ideas.

1. Purpose

Every meeting should have a purpose and end-goal. Objectiveless meetings (meeting just to meet) are not only pointless (literally) but are a drain on time, morale and other resources which could be more productively devoted to other tasks and mission-critical purposes. Each meeting should have a focus and purpose. Ideally, there should only be one central or core objective for the meeting. Never call or organize a meeting without knowing what you seek to accomplish in that session. It is inherently important to know why you are scheduling a meeting.

Does the meeting really need to be held? Having fewer (but better) meetings can provide a breakthrough for increasing productivity, participation when you do hold a meeting and overall satisfaction. Schedule a meeting only when it is necessary for the purpose and objective. Before scheduling a group meeting, ask yourself whether you can achieve your goal in some other way, perhaps through a one-on-one discussion with someone, a telephone conference call, or a simple exchange of emails.

All too often, “meetings” are held to merely update or passively share informational items. If you leave a meeting without having had discussion and interaction and/or any post-meeting action steps, you should question the value of the meeting. A meeting to “share updates” should be replaced with a memo, website update, bulletin board posting, email or voicemail message.
Sue Shellenberger wrote in “The Plan to End Boring Meetings” (Wall Street Journal, 12-21-2016, A11) that managers often invite too many people to attend meetings, as well as ask people to attend the meetings for the wrong reasons resulting in far too many oversized groups that fail to work together effectively. She suggested that the number of meeting participants should be adjusted based on the core purpose of the meeting. Doing so, she argues, may lead to faster and better decisions as well as more engaged employees. Here are her categories of recommendations for number of participants to ask to attend each type of meeting for maximum positive outcomes:

Weighing a problem meeting – 4 to 6 People

Invite enough people to bring needed expertise, without including so many that discussion files off course Each participant should have a role to play….

Making a decision – 4 to 7 people

….For every additional participant over seven, the likelihood of making a sound decision goes down by 10%. According to Michael Mankins, a partner at Bane and Co. “By the time, you get to 17 people, the changes of your actually making a decision are zero….”

Setting the agenda – 5 to 15 people

Another kind of meeting, the daily agenda-setting session, should be brief and vary in size, based on how big your team is. These brief gatherings, often called huddles or stand-up meetings, usually involve only the people who have a logical reason to be there because their work or cooperation is critical to the day’s agenda.

Brainstorming – 10 to 20 people

Sprinkle the list of invitees with people from different backgrounds and social networks to spark diverse ideas…..Brainstorming participants tend to resist throwing out risky or novel ideas because they’re worried about what others might think….[One expert] suggests giving participants time in advance to write down ideas and submit them anonymously before the meeting.

2. Plan

Every meeting should have a plan, developed in advance that answers the basic who, what, when, how and why questions about the meeting. It should lay out how these are communicated in advance to attendees as well as how these will be accomplished in the meeting itself. Perhaps giving a meeting sub-titles could help achieve the goal of sharing the plan for the meeting (e.g. a brainstorming meeting or a timeline development meeting, etc.). include in the plan the method and message of calling the meeting or inviting (requiring) attendance. Attendees should understand in advance “why” they are part of the meeting participants and what to expect in terms of time commitment, preparation or resources to have ready.

Picture
Review previous meetings and previous feedback as you plan the meeting. Learn how to improve meetings by reviewing past presentations and identifying aspects to discard or incorporate into the meeting plan for the next meeting.

Anticipate how much time is required to accomplish the purpose and objectives of the meeting. Set the start and stop time accordingly. Select a meeting space (room) that has appropriate work space (table, sufficient chairs, necessary A/V equipment, etc.). Be prepared. Meetings are work, so, just as in any other work activity, the better prepared you are for them, the better the results you can expect.

There may be certain times (and days) during the week that work best for the meeting. Try to systematically analyze and anticipate the “better” days and times for scheduling a meeting based on the invited attendee’s assignments, duties and other work conflicts. If you use a software calendar – scheduler (e.g. MS Outlook) – don’t schedule a meeting to occur the next minute after a previous meeting or appointment is scheduled to end. This is impractical and unrealistic. Depending on the organizational culture, it may be normative for meetings to routinely “run long” so that some sense of transition time is reasonable. As you schedule meetings, building in 15 minutes of travel time between meetings can be helpful, particularly if attendees are coming from different floors in the building, different buildings or even different campuses to attend the meeting.

3. Descriptive Agenda

Write a one-page summary of the purpose and plan for the meeting. Before the meeting begins, share the one-page summary of the major points that you want to cover during your meeting. This enables employees to know what is expected from them, helps keep the meeting on-track and consistent with the purpose and plan, and results in employees having a better understanding of what to expect in the meeting. Also, this will help reduce any anxieties or fears among your workers and prevent any rumors from spreading before the meeting begins.

An agenda can play a critical role in the success of any meeting. It shows participants where the meeting is going. It is usually best to distribute the agenda and any preparation assignments in advance of the meeting. Research has found that “mystery” meetings tend to have lower levels of motivation, participation and satisfaction than focused and purposeful meetings where the participants understand the why and what expectations of the meeting.

The descriptive agenda should not be an itemized list of specific points but rather a general overview that help set the tone and expectations of what is about to occur. In fact, the person running the meeting may want to have a very detailed personal agenda for the meeting but the one distributed publicly should be general in orientation not detail specific.

4. Start the Meeting

It is important to signal that the meeting has begun. Have a formal threshold to let everyone know that the context has shifted into “meeting mode” and that other activities and small talk should cease and attention turned to the agenda of the meeting. Start the meeting on time. There are many reasons to do this, including: making best use of limited time, signals that the meeting is important, sets expectations about the business purpose of the meeting, overtime tends to encourage prompt attendance and sets up the expectation that the meeting will end on time as well.

5. Participation

One of the core concepts of a meeting is that there is a level of synergy possible which is potentially greater than a single person or two-people considering the topic, subject or idea of the meeting. Thus, it is essential to create a climate to foster engagement, participation and interaction for a meeting to be truly successful. This requires setting aside sufficient time as well as a skillful facilitator to lead the meeting with a goal of fostering participation.
State explicitly that the goal of the meeting is to achieve participation. Establish ground rules that empower all viewpoints to be expressed. Focus on the substantive discussion and topic at hand, not on personalities, procedures or distractions.

6. Facilitation

A meeting leader should function as a facilitator. One tactic business leaders use to avoid inadvertently dominating a meeting is by delegating meeting leadership. Consider whether it is advantageous to assign the meeting management responsibility to someone else, perhaps to build subordinates skills. Other leaders tend to rotate the meeting leader position to other staff in subsequent meetings, which will help them improve their management skills. The delegated personal should have some training and experience in facilitation. Here are a few guidelines which facilitators should consider:
  • Keep the meeting on topic. When there are an abundant amount of people in a meeting, it can be difficult to stay on topic. Prepare accordingly. If you find that the meeting isn’t going anywhere or someone is off on a tangent, politely circle back to the important topic that needs to be addressed. Meetings can easily get off track and stay off track. The role of the facilitator is to keep the meeting on track.
  • Ask Useful Questions. To prepare, write a list of questions that relates to the purpose and objectives of the meeting. If you ask a question and no one answers it, make sure you ask for clarification or push to get an answer that keeps the focus on the subject matter issues.
  • Verbally reward participation. Compliment and express appreciation for those who are engaging and helping advance the goal and purpose of the meeting discussion.
  • Provide constructive feedback for those who are not engaging and advancing the purpose of the meeting. Manage the participants trying to dominate the meeting. Do not let a few people take control of your meetings. Instead, create a friendly atmosphere where everyone feels comfortable expressing their opinions.
  • Allow for sufficient Wait-Time. Sometimes interactive discussion is slow to start (other times the opposite is the challenge). When you ask a question, don’t shortly thereafter answer it yourself. A silent pause in the room may feel awkward but research and experience has demonstrated that that conversation gap is best filled by one of the participants who, after hesitation, begins to offer input.
  • Vary the meeting format. Incorporate some variety in the meetings and do not do the same thing the same way all the time. Be flexible when asking for, receiving and considering suggestions on improving meetings. One interesting technique which I have read about are “standing meetings,” which are meetings in which people gather and remain standing rather than being seated around a conference table. The rationale is that frivolous distractions and longwinded speeches are less likely and there may be grater motivation to focus on the issues and create the action-item lists. If you want a meeting to be short and efficient, a standing meeting might be an option. Another format which I have observed is the “walking meeting.” The walking meeting is conducted while the group “walks.” It has similar advantages to the “standing meeting” but with the added bonus of visiting a work station, lab, classroom or production line on-site is that is appropriate for the purpose of the meeting.

Picture
7. Closure Communication

Don’t end the meeting right away when the discussion wanes. Don’t just finish saying what you want to say and then leave. Typically, there are still good ideas yet unspoken that can be elicited or someone may not fully understand how their idea fits the needs of the meeting. Again, be patient and persistent even if attendees are quiet.

At the end of every meeting, go around and review the action steps each person has captured. Some facilitators also ask for closure observations about the meeting itself or next steps needed to advance the meeting goal and objectives. The exercise takes a small amount of time per person, and is usually well worth the summative feedback.

8. Action Item List

Create an action item list of specific follow-up steps and measures that have arisen during the meeting. Do not assume that everyone will informally remember the various action items slated for follow through. Assign a person’s name as the responsible person to accomplish the action item. This better ensures accountability that it will be accomplished.
It is also helpful to produce notes or “minutes” from each meeting. Don’t just assume that all participants are going to take their assignments to heart and remember all the details. Instead, be sure that someone has agreed to take on the job of record keeping. Immediately after the meeting, summarize the outcome of the meeting, as well as assignments and timelines, and email a copy of this summary to all attendees.

9. End the Meeting

Start on time and end on time. Everyone has suffered through meetings that went way beyond the scheduled ending time. That situation would be fine if no one had anything else to do at work. But in these days of faster and more flexible organizations, everyone always has plenty of work on the to-do list. If you announce the length of the meeting and then stick to it, fewer participants will keep looking at their watches, and more participants will take an active role in your meetings.

10. Assessment

Get feedback. Every meeting has room for improvement. Typically, you want to capture two types of feedback, so structure your data collection methods accordingly. Summative feedback focuses on evaluating the meeting and all aspects of it as it unfolded. This is evaluative feedback from meeting attendees on how the meeting went right for them — and how it went wrong. Was the meeting too long? Did one person dominate the discussion? Were attendees unprepared? Were the items on the agenda unclear? Formative feedback focuses on changes to the meeting plan, procedures and processes that should be implemented or adapted for the next (future) meeting to be held. These would include suggestions for improvements or changes to the way that things were or have been done in the past.

Summary

Time is a precious resource, and it is too expensive to needlessly waste it. With the amount of time devoted to business meetings in organizations, it is appropriate to focus on improving the productivity, participation and satisfaction with meetings.
These are just a few suggestions for improving the performance of the time spent in meetings. It is a win-win opportunity to improve productivity, gain participation and enhance everyone’s satisfaction levels.

0 Comments

(Mis)communication And Deadly Medical Errors

1/12/2017

0 Comments

 
Picture
One area where communication failures and breakdowns too frequently put lives at risk is found in the issue of medical errors in providing health care.

Per research published in the Journal of Health Care Finance (Andel, Davidow, Hollande, and Moreno, 2012[i]) approximately 200,000 Americans die from preventable medical errors including facility-acquired conditions and millions may experience health care provider errors. Medical errors in the United States have an annual impact of roughly $20 billion. Most of these costs are directly associated with additional medical cost, including: ancillary services, prescription drug services and inpatient and outpatient care. Citing previous research (by the Society for Actuaries and conducted by Milliman in 2010) additional costs were attributed to increased mortality rates with about $1.1 billion or more than 10 million days of lost productivity from missed work based on short-term disability claims.

The published research estimated that the economic impact may be much higher when indirect costs are quantified, perhaps another $1 trillion annually when quality-adjusted life years (QALYs) are applied to those that die. Andel et. al, using the Institute of Medicine’s (IOM) estimate of 98,000 deaths due to preventable medical errors annually in its 1998 report, To Err Is Human, and an average of ten lost years of life at $75,000 to $100,000 per year, conservatively projected a loss of $73.5 billion to $98 billion in QALYs for those resulting deaths. Some research suggests that preventable health care error death costs may be up to ten times the IOM estimate.

Quality care and patient safety depends on multiple factors, all of which must be working harmoniously to ensure delivery. However, one key factor is the quality of communication at various critical points in the health care provider sequence. According to the University of Minnesota’ TAGS[ii] site “ Upwards of 100,000 deaths occur in the United States each year because of medical mistakes. One of the biggest factors contributing to the problem is miscommunication or lack of communication between multiple health care professionals.”

The Joint Commission Center for Transforming Healthcare[iii] reports that “ineffective hand-off communication is recognized as a critical patient safety problem in health care; in fact, an estimated 80% of serious medical errors involve miscommunication between caregivers during the transfer of patients. The hand-off process involves “senders,” those caregivers transmitting patient information and transitioning the care of a patient to the next clinician, and “receivers,” those caregivers who accept the patient information and care of that patient. In addition to causing patient harm, defective hand-offs can lead to delays in treatment, inappropriate treatment, and increased length of stay in the hospital.”

Marie McCullough’s writing in The Philadelphia Inquirer[iv] describes a horrific problem of problematic labeling and miscommunication leading to fatal errors in the administration of chemotrophic drugs. She begins the writing with a personal narrative about a patient who died due to a recurring medical error. Christopher Wibeto was receiving vincristine to treat cancer. Vincristine is a chemotherapy medication commonly used to treat several types of cancer. As McCullough describes, shortly after health care providers had injected the drug into Wibeto’s spine, doctors realized that a catastrophic medical error had occurred. Since vincristine is neurotoxic it must be diluted and given intravenously – it should never be injected directly into spinal fluid (which flows around the brain). As a result, Wibeto developed dementia, paralysis and died within days as a result of the improper administration of the drug. McCullough writes that “no one knows how many vincristine disasters have occurred. The Institute for Safe Medical Practices in Horsham has documented 125 fatal misadministrations since the 1960s, but experts believe that the real number is many times higher.” This type of medical error is preventable. Simple changes in the dispending containers, more effective warnings and successful health communication about procedures and verification processes are obviously warranted.  Stories of medical errors such as the Wibetor case simply should never happen.

Most Common Root Causes of Medical Errors

The U.S. Department of Health and Human Services Agency for Healthcare Research and Quality[v] identified a diverse group of factors that cause medical errors. At the top of the list were the factors of communication, information flow, coordination and communication training/planning. Here are the top eight identified factors in rank order:
  1. Communication problems represent the most common cause of medical errors noted by the error reporting evaluation grantees. Communication problems can cause many different types of medical errors and can involve all members of a health care team. Communication failures (verbal or written) can take many forms, including miscommunication within an office practice as well as miscommunication between different components of the health care system or health care providers working different shifts. These problems can occur between health care providers such as primary care physicians and emergency room personnel, attending physicians and ancillary services, and nursing homes and patient services in hospitals. Communication problems can result in poorly documented or lost information on laboratory results, diagnostic testing, or medication information, and can occur at any point along the communication chain. Communication problems can also occur within a health care team in one location, between providers at different locations, between health care teams and other non-clinician providers (such as labs or imaging centers), and between health care providers and patients.
  2. Inadequate information flow can include problems that prevent:
    1. The availability of critical information when needed to influence prescribing decisions.
    2. Timely and reliable communication of critical test results.
    3. Coordination of medication orders at points of interface or transfer of care.
    4. Information flow is critical between service areas as well as within service areas in health care. Often, necessary information does not follow the patient when he or she is transferred to another service or is discharged from one component or organization to another.
  3. Human problems relate to how standards of care, policies, or procedures are followed. Problems that may occur include failures in following policies, guidelines, protocols, and processes. Such failures also include sub-optimal documentation and poor labeling of specimens. There are also knowledge-based errors where individuals do not have adequate knowledge to provide the care that is required for any given patient at the time it is needed.
  4. Patient-related issues can include improper patient identification, incomplete patient assessment, failure to obtain consent, and inadequate patient education. While patient related issues are listed as a separate cause by some reporting systems, they are often nested within other human and organizational failures of the system.
  5. Organizational transfer of knowledge can include deficiencies in orientation or training, and lack of, or inconsistent, education and training for those providing care. This category of cause deals with the level of knowledge needed by individuals to perform the tasks that they are assigned. Transfer of knowledge is critical in areas where new employees or temporary help is often used. The organizational transfer of knowledge addresses how things are done in an organization or health care unit. This information is often not communicated or transferred. Organizational transfer of knowledge is also a critical issue in academic medical centers where physicians in training often rotate through numerous centers of care.
  6. Staffing patterns/work flow can cause errors when physicians, nurses, and other health care workers are too busy because of inadequate staffing or when supervision is inadequate. Inadequate staffing, by itself, does not lead directly to medical errors, but can put health care workers in situations where they are much more likely to make an error.
  7. Technical failures include device/equipment failure and complications or failures of implants or grafts. In many instances equipment and devices such as infusion pumps or monitors can fail and lead to significant harm to patients. In many instances, inadequate instructions or poorly designed equipment can lead to patient injury. Often technical failure of equipment is not properly identified as the underlying cause of patient injury, and it is assumed that the health care provider made an error. A complete root cause analysis often reveals that technical failures, which on first review are not obvious, are present in an adverse event.
  8. Inadequate policies and procedures guiding the delivery of care can be a significant contributing factor in many medical errors. Often, failures in the process of care can be traced to poorly documented, non-existent, or clinically inadequate procedures.

Research at Stanford Medicine concludes that “better communication between caregivers reduces medical errors[vi].” The Stanford research found that focused efforts to improve communication quality alone resulted in a 30% decline in preventable adverse medical error events. (A copy of that research and experimental program is available for download at http://www.ipasshandoffstudy.com.) One of the important benefits from communication education and training is the reduction of preventable medical errors due to communication problems, failures and breakdowns. It is well past time that communication become a priority for health care professionals.

[i] Andel C, Davidow SL, Hollander M, Moreno DA., (2012) The economics of health care quality and medical errors. J Health Care Finance, 2012 Fall;39(1):39-50.
[ii] http://www.healthtalk.umn.edu/2014/04/11/preventing-medical-miscommunication-means-fewer-medical-errors/
[iii] http://www.jointcommission.org/assets/1/6/TST_HOC_Persp_08_12.pdf
[iv] McCullough, Marie (2016) Fighting a deadly chemo error, The Philadelphia Inquiry, November 11, 2016, A2.
[v] https://archive.ahrq.gov/research/findings/final-reports/pscongrpt/psini2.html
[vi] https://med.stanford.edu/news/all-news/2014/12/better-communication-between-caregivers-reduces-medical-errors.html


0 Comments

The Coming Leap Second Clock Adjustment: A Little Prevention is Worth The Effort

1/4/2017

0 Comments

 
Picture
For those of you who remember the disaster mitigation and recovery planning that was associated with the Y2K technological risk, you might find it interesting to note that small adjustments in the world’s atomic clocks still need to be done periodically to keep us all in sync and as well maintain social and economic harmony with the planet. Most people are aware of “Leap Year” adjustments where an entire day is added to the year (in the month of February) but fewer know that there are also “Leap Second” adjustments inserted periodically as well.

Atomic Time vs. Universal Time


Coordinated Universal Time or UTC is a standard, not a time zone. In other words, it is the base point for all other time zones in the world. They are determined by their difference to UTC. UTC is represented as UTC +0. Coordinated Universal Time is a 24-hour time standard that is used to synchronize world clocks. In order to keep Coordinated Universal Time as accurate as possible, two other time standards are used: International Atomic Time or TAI, and Universal Time also known as Solar Time.

There are two components used to determine Coordinated Universal Time (UTC). These are:
  1. International Atomic Time (TAI): A time scale that combines the output of some 200 highly precise atomic clocks worldwide, and provides the exact speed for our clocks to tick.
  2. Universal Time (UT1), also known as Solar or Astronomical Time, refers to the Earth’s rotation around its own axis, which determines the length of a day.

As you might expect, these two units of measurement gradually move out of synchronization with each other. When the difference between UTC and UT1 approaches 0.9 seconds, a leap second is added to UTC and to clocks worldwide. By adding an additional second to the time count, our clocks are effectively stopped for that second to give Earth the opportunity to catch up with atomic time. The reason we have to add a second now and then is that Earth’s rotation around its own axis is gradually slowing down, although very slowly. Atomic clocks, however, tick away at pretty much the same speed over millions of years. Compared to the Earth’s rotation, atomic clocks are simply too consistent.

Upcoming leap seconds are announced by the International Earth Rotation and Reference System Service (IERS) in Paris, France. Before the first leap second was added in 1972, UTC was 10 seconds behind Atomic Time. So far, a total of 26 leap seconds has been added. This means that the Earth has slowed down an additional 26 seconds compared to atomic time since then. (However, this does NOT mean that the days are 26 seconds longer nowadays. The only difference is that the days a leap second was added had 86,401 seconds instead of the usual 86,400 seconds.)

Leap seconds and leap years are both implemented to keep our time in accordance with the position of Earth. However, leap seconds are added when needed, based on measurements, and leap years are regularly occurring events based on set rules. During leap years, an extra day is added as February 29th to keep the calendar synchronized with the precession of the Earth around the Sun. Leap years are necessary because the actual length of the year is 365.2422 days and not 365. The extra day is added every four years to compensate for most of the partial day. However, this is a slight over-compensation, so some century years are not leap years. Only every fourth century year (those equally divisible by 400) is a leap year. For instance, 2000 was a leap year, but 1900, 1800 and 1700 were not.

Business Continuity Implications


The next leap second will be added on December 31, 2016 at 23:59:60 UTC. The difference between UTC and International Atomic Time (TAI) will then increase from the current 36 seconds to 37 seconds.

According to the National Institute of Standards and Technology (NIST) by keeping Coordinated Universal Time (UTC) within one second of astronomical time, scientists and astronomers observing celestial bodies can use UTC for most purposes. If there were no longer a correction to UTC for leap seconds, then adjustments would have to be made to time stamps, legacy equipment and software which synchronize to UTC for astronomical observations. However, adding a second to UTC can create problems for some systems, including data logging applications, telecommunication systems and time distribution services. Special attention must be given to these systems each time there is a leap second.

I recently ran across Tom Brant’s essay about the coming adjustment and some of the technological implications in PC Magazine “2016 Needs a ‘Leap Second’ to Sync With Earth’s Rotation” in which describes the next “leap second” adjustment to Atomic Clocks scheduled to occur on December 31, 2016.

According to Brant, “the people responsible for measuring the world’s time have got very good at determining just how long a second is supposed to last—the most accurate clock in the world uses cesium atoms to determine the exact length of a second, and it won’t get out of sync for at least 300 million years. But the question of how many seconds are in a year is far less certain. The International Earth Rotation and Reference Systems Service, which is responsible for Coordinated Universal Time (UTC), decides twice a year whether or not a “leap second” is needed to ensure that the world’s clocks are in sync with the Earth’s rotation, and this week its scientists decided that 2016 needs one of those extra seconds. It will be added at midnight on Dec 31, when clocks will read 11:59:59 p.m., then 11:59:60 p.m., before the stroke of 12:00:00 a.m. ushers in the year 2017. That slowdown is roughly equivalent to a loss of around two milliseconds per day, so the Paris-based IERS evaluates whether or not to add a leap second twice per year, on June 30 or December 31. As the US Naval Observatory explains, ‘[a]fter 500 days, the difference between the Earth rotation time and the atomic time would be one second. Instead of allowing this to happen a leap second is inserted to bring the two times closer together.’ A leap second has been added 26 times since the practice began in 1972, according to the observatory.”

“One day this year we’ll have 86,401 seconds, not the usual 86,400. When that’s happened before it’s caused some software to get way out of whack.”
– Network World

Patrick Nelson’s writing in Network World before the last Leap Second adjustment identified some of the IT business continuity concerns:

“…official clocks will pause by one second to let the earth’s rotation catch up with atomic time. Shouldn’t be a problem, right? Only tell that to LinkedIn, Reddit, and Qantas. All three were running systems that crashed in 2012, when the last leap second was added. The prior leap second in 2005 also caused problems with some computers, including Google’s. Well, it’s time again for another one. So brace yourself for potential trouble. Indeed, it may well be time to ask server system vendors about their mitigation plans….


What happened in 2012? Issues arose at Foursquare, LinkedIn, Mozilla, Qantas, Reddit, StumbleUpon and Yelp, who all reported crashes, according to media reports. Joab Jackson of IDG News Service wrote at the time that unpatched Linux kernels, Hadoop instances, Cassandra databases and Java-based programs were affected. Some servers running Debian Linux went offline. What caused it?


Computing systems and their Network Time Protocol, or NTP, client software need to be programmed to handle unforeseen extra seconds. If the software isn’t programmed correctly, unexpected seconds can cause problems. NTP is used to sync with the atomic clock. In some cases in the 2012 leap second implementation, NTP had to be disabled in order to restore servers. Linux patches were available before that leap year adjustment because the NTP high-resolution timer used was known to potentially cause a livelock. Livelocks are a way that a process doesn’t progress. The patches presumably weren’t applied in some cases.”


Robert McMillan’s writing in advance of last year’s leap second insertion in Wired.com (“The Leap Second Is About to Rattle the Internet. But There’s a Plot to Kill It”) reported:

“The Qantas Airways computers started crashing just after midnight. A few hours later, as passengers started flying home from weekend getaways, there were long delays in Brisbane, Perth, and Melbourne, and the computers still didn’t work. Qantas flight attendants were forced to check passengers in by hand. That Sunday morning in July 2012 was a disaster for Amadeus IT Group, the Spanish1 company responsible for the software that had computer screens flickering at Qantas kiosks. But it wasn’t entirely the company’s fault. Most of the blame lay with an obscure decades-old timing standard for the UNIX operating system, a standard fashioned by well-intentioned astronomer time lords. They were working for an international standards body, a precursor of the International Telecommunications Union, which today officially tells clock-keepers how to tell the rest of the world what time it is. Back in 1972, they decided to insert the occasional leap-second into Coordinated Universal Time (UTC), the standard most of the world uses to set wristwatches.


We’ve had 25 of these leap seconds since then, and we’re about to get our 26th. This week, the modern time lords announced that the next leap second will arrive at 11:59 pm and 60 seconds on June 30. That has some computer experts worried. Amadeus wasn’t the only company to go glitchy during the last leap-second. Reddit, Foursquare, and Yelp all blew up thanks to the leap second and the way it messed with the underlying Linux operating system, which is based on UNIX.


The trouble is that even as they use the leap second, UNIX and Linux define a day as something that is unvarying in length. ‘If a leap second happens, the operating system must somehow prevent the applications from knowing that it’s going on while still handling all the business of an operating system,’ says Steve Allen, a programmer with California’s Lick Observatory. He likens it to the problem facing the HAL 9000, the fictional onboard computer in Stanley Kubrick’s 2001: A Space Odyssey, which loses its mind after it is programmed to lie. ‘All the problems that crop up are, in a metaphorical sense, the HAL 9000 problem. You have told your computer to lie. I wonder what it will do,’ he says. The Linux kernel folks aren’t expecting any major issues when July 1 comes around, but the situation is unpredictable. Back in 2012, Linux creator Linus Torvalds told us: ‘Almost every time we have a leap second, we find something.’ And this time around, there will be problems again. Torvalds doesn’t think they’ll be as widespread as they were three years ago, but they’re largely unavoidable. The ‘reason problems happen in this space is because it’s obviously rare and special, and testing for it in one circumstance then might miss some other situation,’ he says.”


The leap second is yet another uncertainty for which business continuity planners, and particularly those responsible for IT systems, should be aware and take preparedness measures to mitigate and/or quickly recover from the next predictably unforeseen event.

View the following PBS video to learn more about the science behind the Leap Second.


0 Comments

    dR. Robert Chandler

    Contact

    Archives

    January 2017
    December 2016
    November 2016
    October 2016
    September 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    March 2016
    February 2016
    January 2016
    December 2015
    November 2015
    October 2015
    September 2015
    August 2015
    July 2015
    June 2015

    Categories

    All
    Business Continuity Planning
    Crisis Communication
    Crisis Management Leadership
    Crisis Management Teams
    Dr. Robert Chandler Articles
    Health And Risk Communication
    Human Performance Factors

    RSS Feed

    View my profile on LinkedIn
Copyright 2016, Dr. Robert Chandler   |   Contact