Home Articles What Happened At Knight?

What Happened at Knight?

PDF Print E-mail

Wall Street got a dramatic reminder of the value of strong configuration management (CM) stewardship on August 1, 2012, when Knight Capital Group experienced an incident which resulted in the erroneous purchase of over 7 billion dollars in stocks. Knight had little choice but to sell as many of the stocks as possible, resulting in a 440 million dollar loss which I reported on in stickyminds. Bloomberg published an article on August 14th that claims that the software glitch was due to, "software that was inadvertently reactivated when a new program was installed, according to two people briefed on the matter." The Bloomberg articlewent on to say, "Once triggered on Aug. 1, the dormant system started multiplying stock trades by one thousand, according to the sources, who requested anonymity because the firm hasn’t commented publicly on what caused the error. Knight’s staff looked through eight sets of software before determining what happened, the people said." This incident highlights the importance of Configuration Management Best Practices. This article will describe some of the essential IT controls that could have potentially prevented this mistake from occurring (and I personally guarantee that they would cost a lot less than 440 million dollars to implement). First, here's a quick description of the regulatory environment within which Knight Capital and other financial services firms must operate on a daily basis.

Background:

Federal regulatory agencies, including the Federal Financial Institutions Examination Council's (FFIEC) and the Office of the Comptroller of the Currency (OCC), monitor the IT controls established by banks and other financial institutions who are required to comply with section 404 of the Sarbanes-Oxley Act of 2002. The ISACA Cobit framework is the defacto standard which describes the essential controls that need to be established  - including both change and configuration management.
 

Taking Control:

In a regulatory environment, all changes need to be controlled. This is usually accomplished by having the proposed changes reviewed by a Change Control Board (CCB). Each request for change (RFC) must be reviewed with an assessment of the potential downstream impact (e.g. risks) of the change. Once approved, releases can be deployed and then verified. In CM terminology, there is a requirement for a physical and functional configuration audit, which means that the deployed binaries (called configuration items) must be verified to be the correct version and also that they are functioning as desired. Obviously, software must be thoroughly tested before it is approved for promotion to production.

Automating the application build, package and deployment is essential for success and this is precisely what DevOps is all about. In classic CM terminology, status accounting is the function that tracks a configuration item (CI) throughout its lifecycle and this would absolutely include retiring (uninstalling) any assets determined to be no-longer needed. 


Apparently, Knight Capital lacked the necessary procedures to accurately track changes and deploy their code. According to the Bloomberg report, Knight did not know exactly what code had been deployed to their production servers and, most importantly, how to retire assets that were no longer being utilized.

Now before anyone starts to feel too smug, let's consider the fact that most of the financial services firms on Wall Street lack the basic configuration management procedures to ensure that this same problem cannot occur on their servers. Financial services firms are not the only companies lacking CM Best Practices. I have also seen medical firms (including those rsponsible for surgical operating room equipment), government agencies and many other companies with mission critical systems that lack these basic competencies.

It's time for IT controls to be implemented on all computer systems that matter. This just makes sense and obviously improves productivity and quality. Not to mention the savings benefit - proper CM controls can prevent many types of errors which, though easily overlooked, can cause millions of dollars of losses in just minutes!

Bob Aiello
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
twitter: @bobaiello, @cmbestpractices
http://www.linkedin.com/in/BobAiello

 

 

 



More articles by this author

Micro Focus Completes Acquisition of Serena Software, Inc. Application Lifecycle Management Acquisition Boosts Micro Focus’s DevOps Capability ROCKVILLE, Md., May 2, 2016 /PRNewswire/ -- Micro Focus (LSE: MCRO.L) today announced the completion of its acquisition of Serena Software, a leading provider of Application Lifecycle Management (ALM) software, under the terms of the definitive agreement disclosed on March 22, 2016. "Our customers continue to look at DevOps as a way to deploy critical applications and services quickly and with greater reliability to meet business demands," said Stephen Murdoch, CEO, Micro Focus. "The Serena acquisition extends our ability to help customers meet these challenges so they can drive greater innovation faster with lower risk." According to industry analyst firm Gartner, "DevOps implementations utilize technology, especially automation tools, that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective."1  The experience and expertise which the Serena business brings will enable Micro Focus to help its customers develop and release applications and services faster, with greater speed and accuracy. Serena adds capabilities in software application development; software configuration and change management; and business process management to Micro Focus's portfolio of ALM solutions spanning mainframe environments, distributed systems and cloud. The combination of Micro Focus and Serena allows companies to better: Design and build business applications and services with greater accuracy, reliability and predictability; Continuously deploy existing core business applications on a wider variety of platforms to meet changing business needs; and Improve the speed and efficiency of new business services through automated release and deployment solutions. About Serena Software Serena is among the largest Application Lifecycle Management vendors with more than 2,500 enterprise customers. Serena helps the highly regulated large enterprise move fast without breaking things – increasing velocity of the software development lifecycle while enhancing security, compliance, and performance. More information is available at www.serena.com. About Micro Focus Micro Focus (LSE: MCRO.L) is a global enterprise software company helping customers innovate faster with lower risk. Our software helps customers build, operate and secure IT systems that bring together existing business logic and applications with emerging technologies to meet increasingly complex business demands. For more information, visit: www.microfocus.com. 1I&O Must Combine ITIL and DevOps to Deliver Business Value for Bimodal IT," by George Spafford and Ian Head, March 18, 2016.
Hi, I am excited to invite you to subscribe to our new online publication which provides guidance on Configuration Management Best Practices, Agile Application Lifecycle Management (including videos) and, of course DevOps. Our publication explains hands-on best practices required to implement just enough process to ensure that you can build software and systems that are reliable and secure. Based upon our new book, Agile Application Lifecycle Management - Using DevOps to Drive Process Improvement, the Agile ALM DevOps Journal seeks to promote a dialogue that is really needed in the industry today. We will discuss practical approaches to implementing the Agile ALM using DevOps best practices including continuous integration, delivery and deployment. We will also discuss process improvement strategies that work in large organizations that must comply with federal regulatory guidelines, along with smaller teams that may very well grow as they become successful. We are taking this journey together and our goal is to ensure that you have a voice and can share your experiences along with learning from other colleagues too. Enjoy Leslie Sachs's amazing column: Personality Matters and Bob Aiello's column: Behaviorally Speaking. We will also report on major incidents where organizations clearly need to improve on their ability to develop and deliver software effectively, including the recent Southwest systems glitches which resulted in thousands of flights being cancelled and thousands more being delayed. We will also bring you exciting technical product announcements such as jfrog's new xray, which helps to scan your runtime objects, including docker images, for vulnerabilities. This is an exciting time to be in the technology field. Join the revolution! You can submit your articles for publication to share your own knowledge and experience!  Subscribe to receive your copy and register so that you can comment online Bob Aiello http://www.linkedin.com/in/BobAiello @bobaiello, @agilealmdevops, @cmbestpractices This e-mail address is being protected from spambots. You need JavaScript enabled to view it  
 
Copyright © 2017 CM Best Practices. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.
 

Product News

Live Online Training

Jobs