Please contact Jonathan Bowen if you know of relevant online information not included here.

---

This document contains pointers to information on Safety-Critical Systems, where human lives may be at risk, especially involving software and computers, available around the world on the World Wide Web (WWW).

indicates new information. indicates (subjectively) recommended information.

---

Contents 1 Relevant newsgroups 2 Repositories and mailing lists 3 Courses 4 Clubs and organizations 5 Publications 6 Journals 7 Books 8 Companies 9 Meetings

Relevant newsgroupsEdit

• The ACM Forum on Risks and the associated moderated newsgroup comp.risks is recommended reading. Also available on the Web from Google Groups. See a count of and link to URLs mentioned in RISKS articles from People Helping One Another Know Stuff (PHOAKS). You can join the gatewayed RISKS mailing list (only if you cannot access the newsgroup for some reason please!) by sending an email request to risks-request@csl.sri.com. Submissions for possible inclusion may be sent to risks@csl.sri.com. A RISKS archive in the UK is also available, including the latest issue and searching. Submissions are sometimes included in the ACMSIGSOFT Software Engineering Notes. See associated book.
• Discussion on safety-critical systems also crops up in comp.software-eng from time to time. See also Google group interface.
• The sci.aeronautics.airliners newsgroup often includes discussion and information on aircraft safety. See also Google group interface.
• The sci.engr.safety newsgroup on the Safety of Engineered Systems has been established by Sethu Rathinam rathinam@netins.net (see RISKS DIGEST 16.53, November 1994). This was passed by 334 votes to 11 (20 March 1995). See also Google group interface.
• The misc.industry.safety.personal for discussion on personal protective equipment (PPE) - any device worn or held by the user in order to provide protection whilst performing the given task. See also Google group interface.
• Other articles on safety-critical systems from Google groups.
• Search for websites on safety-critical system from Google.

Repositories and mailing listsEdit

• A BibTeX bibliographic database of publications of relevance to safety-critical systems (second-sourced in the US).
• NASA Langley formal methods program including many applications to safety-critical systems.
• At the Software Engineering Institute, Carnegie-Mellon University, there is an archive of information relevant to dependable software including a README file. There is an associated mailing list. Contact Charles Weinstock on weinstoc@sei.cmu.edu for further information. Messages for the list should be sent to depend-sw@sei.cmu.edu.
• Jonathan Moffett (jdm@minster.york.ac.uk) of the High Integrity Systems Engineering Group (HISE), Department of Computer Science, University of York, UK, runs a safety-critical mailing list. To join the mailing list, please complete the online form. See also archive of messages. See a moderated discussion on C++ and safety, edited by Brian Wichmann.
• An archive of aviation safety information especially with respect to fly-by-wire aircraft such as the Airbus A320 and the Boeing 777 is available. For example, see the plot of the aviation safety size of software packages of various avionics implementations over the years. This archive is maintained by Robert Dorsett, rdd@netcom.com.
• Since 1989, a moderated mailing list dedicated to all types of safety issues, both occupational and environmental, has been in existence. Subscribe by sending an email message to listserv@uvmvm.uvm.edu with "SUB SAFETY your name" in the body of the message). To post a message, email safety@uvmvm.uvm.edu.
• The Center for High Assurance Computing Systems at Information Technology Division of the Naval Research Laboratory conducts interdisciplinary research and development in techniques for processing and communicating data that preserve critical system properties such as safety.
• The ESPRIT ProCoS and PDCS Basic Research projects investigated techniques concerned with improvements for safety-critical systems.
• The Experimental Computation Laboratory is an organization in the Computer Science Department at the University of Missouri - Rolla dedicated to research in advanced methods in distributed and parallel computation. The current focus is on the use of rigorous mathematics through formal methods to create fault-tolerant and secure real-time distributed computing systems.
• The High Integrity Systems Engineering Group (University of York, UK) undertakes research into all aspects of high integrity computer-based systems, including the ASAM-II project dedicated to the production of a prototype Safety Argument Manager and method.
• Information on Software Risk Management (including the SEI Program on Software Risk Management) and Dependability from the Software Engineering Institute, Carnegie-Mellon University, Pittsburgh, USA.
• ESPRIT Basic Research: PDCS 2 (Predictably Dependable Computing Systems), ProCoS II and ProCoS-WG (Provably Correct Systems).
• The TTM/RTTL framework for real-time reactive programs and devices found in embedded, safety-critical, or concurrent systems.
• High Integrity Software System Assurance (HISSA), NIST, USA.
• NASA Information Services including Goddard Space Flight Center and Johnson Space Center.
• Internet Disaster Information Network - latest news on disaster situations.
• Safety Critical Systems, Safety, Competency and Commitment, Safety-related systems: Guidance for engineers (Hazards Forum) and Functional Safety Professional Network information from the IEE, UK.
• Computer-Related Incidents with Commercial Aircraft by Prof. Peter Ladkin, with hyperlinks to RISKS articles. See also Talking to Newspapers: A Cautionary Tale with Moral.
• Nancy Leveson, Professor of Aeronautics and Astronautics, MIT, USA. A leading software safety expert.
• Air Accidents Investigation Branch (AAIB), Department of Transport, UK. See formal investigations.
• Health and Safety Executive (HSE), UK. See Paddington/Ladbroke Grove Train Crash, 5 October 1999. See also Health and Safety Laboratory (HSL), Sheffield / Buxton, UK.
• ARIANE 5 Flight 501 Failure Report, 19 July 1996, from the European Space Agency (ESA).
• Safety information from NASA Office of Flight Assurance, USA.
• MIL STD 882C, System Safety Programme Requirements, US military safety standard. (See also Appendix A.)
• Aviation Safety from the Federal Aviation Administration (FAA), USA. See also Regulation and Certification Group (AVR), promoting safety standards, including the Office of Accident Investigation and Preliminary Accident and Incident Data.
• National Transportation Safety Board, USA.
• Report to the Defence and Aerospace Foresight Panel, Executive Summary, High Integrity Real-Time Software Working Party, NPL, prepared under the auspices of the BCS.
• Programme for Highly Dependable Systems, Department of Computer Science, University of the Witwatersrand, South Africa.
• UK DTI/EPSRCSafety and Mission Critical Systems Programme information (originally called "SafeIT"). See project descriptions. See also advances in safety-critical systems.
• Glasgow Accident Analysis Group, UK. Developing new means of generating accident reports using formal methods.
• Therac-25 incident.
• National Transportation Safety Board (NTSB), USA. See aviation accident searchable database and statistics.
• Safety Critical Computing Group, School of Computer Studies, University of Leeds, UK.
• Hazop Home Page, School of Computing and Mathematics, University of Teesside, UK. Includes a UK Defence Standards guide.
• White House Commission on Aviation Safety and Security, Department of Transporation, USA. See Final Report in PDF and (long) HTML format. (Completed 12 February 1997.)
• North Texas System Safety Society, USA. Includes software safety links.
• IFIP WG10.4 on Dependable Computing and Fault Tolerance.
• GRS, Germany. A non-profit organization for the assessment and development of the safety of complex technical systems, such as nuclear plants. (In English and German.)
• Aviation Safety Web Pages: Accident Database (1945 onwards).
• Flight Safety Foundation, including publications.
• Transportation Safety Board of Canada (in and French), including [http://bst-tsb.gc.ca/airlist.html aviation occurrence reports and useful links. ]
• Disaster series on BBC2, UK, starting 5 January 1998.
• System safety verification information, including many links, from the formalWARE collaborative project, Canada.
• ESPRESS project, Germany. Engineering of safety-critical embedded systems.
• WorldSafety.com. A global community center for safety and security professionals. See links.
• Dependable System Architecture Group, SRI Computer Science Laboratory, California, USA. See 00-56 Defence Standard information.
• Safety-Critical Sysytems class information, part of a Virginia Tech course.
• Safety Systems Research Centre (SSRC) and Safety Systems Group, University of Bristol, UK.
• Software Safety Planning Group, IEEE Software Engineering Standards Committee (SESC).
• SPARK Ada, high-integrity software development. (An Ada subset.)
• Safety-Critical Software Development using Perfect Developer from Escher Technologies Limited.

See also information on formal methods which are often recommended for safety-critical systems.

CoursesEdit

• Safety Critical Systems course, Software Engineering Programme, Oxford University, UK.
• Modular MSc in Safety Critical Systems Engineering, University of York, UK.
• PES Hazard Study course, School of Computing and Mathematics, University of Teesside, UK.
• Training and technology transfer courses including a short hazard analysis course, Adelard, UK.
• Safety-Critical Systems Development final year undergraduate course by Prof. Chris Johnson, University of Glasgow, UK. Includes online slides.
• Training courses from Mark Bofinger, Savive Pty Ltd, Australia.
• Safety & security critical systems short course, King's College London, UK, 3-7 September 2001.

Clubs and organizationsEdit

• Safety-Critical Systems Club (SCSC), UK. Enquiries should be sent to Mrs. J. Atkinson on csr@ncl.ac.uk (Centre for Software Reliability, University of Newcastle upon Tyne, Newcastle NE1 7RU, UK). The club organizes the annual Safety-critical Systems Symposium and produces a regular newsletter Safety Systems (Editor: Felix Redmill, email felix.redmill@ncl.ac.uk, 22 Onslow Gardens, London N10 3JU, UK; tel & fax: +1-181-883-0789). See also BCS information.
• European Network of Clubs for REliability and Safety of Software (ENCRESS), sponsored by the ESSI Programme of the EC (1994–1997).
• European Safety and Reliability Association (ESRA). General Secretariat: Apartado 1690, 1016 Lisboa Codex, Portugal. Chairman: David Harvey, WS Atkins, Woodcote Grove, Ashley Road, Epsom, Surrey KT18 5BW, UK (email dwharvey@wsatkins.co.uk, tel +44-1372-726140 x4604, fax +44-1372-740055). Associated with the journal Reliability Engineering & System Safety.
• EWICS TC7 (European Workshop on Industrial Computer Systems, Technical Committee 7) on Reliability, Safety and Security. An ESRA Technical Committee since 1995. Contact Robin Bloomfield on reb@adelard.co.uk for further information.
• Safety/Security Club, UK DTI-funded Safety Policy and Models (SPaM) project. Email John Elliott on jje@src1.demon.co.uk for further details.
• IEEE SESC (Software Engineering Standards Committee) Safety Study Group.
• IEEE Society on Social Implications of Technology (SSIT). Includes environment, health and safety issues. See IEEE Technology and Society Magazine.
• American Society of Safety Engineers (ASSE). See related sites including other professional organizations.
• National Safety Council, USA.
• International Safety Council.
• Canada Safety Council, Canada.
• National Safety Council, Ireland.
• Board of Certified Safety Professionals (BCSP), USA.
• System Safety Society, Unionville, VA, USA.
• EUROCONTROL, European organization for the safety of air navigation.
• MISRA, The Motor Industry Software Reliability Association, UK. See Guidelines for the Use of the C Language in Vehicle Based Software.
• Lloyd's Register, shipping safety, UK.
• Safety Regulation Group, Civil Aviation Authority (CAA), UK.
• The Safety and Reliability Society, UK. A professional affiliate of the Engineering Council.

No known online information is available from some of the above organizations. Pointers would be gratefully received.

See also the French National Institute for Transport and Safety Research (INRETS) in English and French.

PublicationsEdit

• Formal Methods in Safety-Critical Standards, Jonathan Bowen. In Proc. 1993 Software Engineering Standards Symposium (SESS'93), Brighton, UK, 30 August - 3 September 1993. IEEE Computer Society Press, pp 168–177, 1993.
• The Ethics of Safety-Critical Systems, Jonathan Bowen. Communications of the ACM, 43(4):91-97, April 2000.
  Based on an invited talk at ENCRESS'97, 3rd International Conference on Reliability, Quality & Safety of Software-Intensive Systems, Athens↑, Greece, 29–30 May 1997.
• Safety-Critical Systems, Formal Methods and Standards, Jonathan Bowen and Victoria Stavridou. IEE/BCS Software Engineering Journal, 8(4):189-209, July 1993. Previously issued as a Programming Research Group Technical Report PRG-TR-5-92. A revised version is available as Chapter 1 in Towards Verified Systems. Winner of the IEE Charles Babbage Premium award, 1994.
• The Industrial Take-up of Formal Methods in Safety-Critical and Other Areas: A Perspective, Jonathan Bowen and Victoria Stavridou. In J.C.P. Woodcock and P.G. Larsen (eds.), FME'93: Industrial-Strength Formal Methods, First International Symposium of Formal Methods Europe, Odense, Denmark, 19-23 April 1993, Springer-Verlag, LNCS 670, pp 183–195, 1993.
• Formal Methods and Software Safety, Jonathan Bowen and Victoria Stavridou. In Heinz Frey (ed.), Safety of Computer Control Systems 1992 (SAFECOMP'92), Proc. IFAC Symposium, Z�rich, Switzerland, 28-30 October 1992, Pergamon Press, pp 93-98, 1992.
• Formal Methods for the Specification and Design of Real-Time Safety Critical Systems, J. Ostroff. Journal of Systems and Software, 18(1):33-60, April 1992.
• Safety-Critical Software: Status Report and Annotated Bibliography, P. Place and K. Kang. Technical Report CMU/SEI-93-TR-5, Software Engineering Institute, Carnegie Mellon University, USA, 1993.
• Formal Specification and Verification for Critical Systems: Tools, Achievements and Prospects, John Rushby (SRI). In Electric Power Research Institute TR-100294, pp 9–1 to 9–14, January 1991.
• Formal Verification of Algorithms for Critical Systems, John Rushby (SRI). IEEE Transactions on Software Engineering, 19(1):113-233, January 1993.
• Formal Methods and the Certification of Critical Systems, John Rushby (SRI). SRI Technical Report CSL-93-7, December 1993. (300 pages) See also CSL-95-1.
• Formal Methods and their Role in the Certification of Critical Systems, John Rushby. SRI Technical Report CSL-95-1, March 1995.
  This is a shorter (50 pages) and less technical treatment of the material in CSL-93-7. It will become a chapter in the FAA Digital Systems Validation Handbook (a guide to assist FAA Certification Specialists with advanced technology issues).
• New Directions in Software Safety: Causal Modelling as an Aid to Integration, Peter Fenelon and John A. McDermid. High Integrity Systems Engineering Group, Department of Computer Science, University of York↑, Heslington, York YO1 5DD, UK. See also other papers and reports by Pete Fenelon.
• Risk management is for everyone, Felix Redmill. IText 1(2), British Computer Society and Oxford University Press, 1995.
• The list of publications held at NASA Langley produced under NASA↑ sponsorship since 1989 by various organizations, together with some online publications may be of interest. An overview of the NASA Langley programme with a good selection of references is available.
• Papers on safety-critical systems in computer science [wais://daneel.rdt.monash.edu.au:210/cs-journal-titles?safety+critical journals] (authors/titles only) and [wais://daneel.rdt.monash.edu.au:210/cs-techreport-abstracts?safety+critical technical reports] (abstracts).
• Software Risk publication abstracts (searchable) from Research Access.
• DOD-STD-2167A Military Standard: Defense system software development.
• How numbers are tricking you by Arnold Barnett in the MIT Technology Review magazine.
• Safety-Critical Systems Computer Language Survey Results by Robin Rowe, posted on comp.software-eng, etc., November 1994.
• Fault Detection, Supervision and Safety for Technical Processes 1994, edited by Ruokonen, Pergamon, 1994.
• Fault Detection, Supervision and Safety for Technical Processes 1997, edited by Ruokonen, Pergamon, 1997. In preparation.
• Safety, Reliability and Applications of Emerging Intelligent Control Technologies, edited by Ng. and Hung, Pergamon, 1995.
• Publications on safety from a searchable catalogue by Elsevier.
• Safety Related Computer Controlled Systems Market Study by Coopers & Lybrand, HMSO, 1992.
• 00-55, 00-56 and other Defence Standards from the UK Ministry of Defence (MOD) Directorate of Standardization. See also 00-56 and Computer Guidance.
• System and Software Safety in Critical Systems, Ulla Isaksen, Jonathan Bowen and Nimal Nissanke. Technical Report RUCS/97/TR/062/A, Department of Computer Science, The University of Reading, UK, 1997.
• Software Safety Standards.
• IEC 61508: Functional safety of electrical/electronic/programmable electronic safety-related systems. See also European SIPI project on Safety in the Process Industries: Guiding Principles to the Implementation of IEC-61508.

Search for and critical publications on safety-critical systems and see the Dependability References Database in The Collection of Computer Science Bibliographies.

See also safety-critical software links from the Online Ethics Center.

JournalsEdit

• Control and Instrumentation, covers hardware and software for process engineering (much of which is safety critical). Editor: Brian Tinham, 30 Calderwood Street, London SE18 6QH, UK (tel: +44-181-855-7777, fax: 0181 316 3422). Assistant Editor: Jeremy Jones, email cz72@cityscape.co.uk.
• High Integrity Systems, published by Oxford University Press, vol 1 nos 1–6, 1994–1996. An excellent forum for papers on safety-critical systems, now sadly ceased publication. Editor: Prof. A.D. McGettrick, email adm@cs.strath.ac.uk.
• Reliability Engineering & System Safety from Elsevier Applied Science, Oxford, is edited by G.E. Apostolakis, School of Engineering and Applied Science, University of California, 38–137 Engineering IV, Los Angeles, California 90024-1597, USA (fax: +1-310-206-2302, no known email address).
• International Journal of Reliability, Quality and Safety Engineering (IJRQSE) from World Scientific.
• International Journal of Risk & Safety in Medicine from Elsevier, Amsterdam.

See also:

• Safety-related journals in the Directory of Computing Science Journals.
• Journals relevant to formal methods.

BooksEdit

The following relevant books have online information:

• Justifying the Dependability of Computer-based Systems, by Pierre-Jacques Courtois, Springer Series in Reliability Engineering, 2008, ISBN 978-1-84800-372-9.

• Safeware: System Safety and Computers, by Nancy G. Leveson. Addison-Wesley, 1995. ISBN 0-201-11972-2, $49.50.
• Computer Related Risks, by Peter G. Neumann, SRI, ACM Press Books (ACM Press / Addison-Wesley), 1995. [1] ISBN 0-201-55805-X, $24.75. See also Risks Forum.
• Software in Safety Related Systems, by Brian A. Wichmann, NPL, Wiley, 1992. ISBN 0471-93474-7.
• Safety-Critical Computer Systems, by Neil Storey, Addison-Wesley, 1996. ISBN 0-201-42787-7.
• High Integrity Ada: The Spark Approach, by John Barnes with Praxis Critical Systems, Addison Wesley Longman, 1997. ISBN 0-201-17517-7. (Includes CD-ROM.)
• Software Safety and Reliability: Techniques, Approaches, and Standards of Key Industrial Sectors, by Debra S. Herrmann. IEEE Computer Society Press, 2000. ISBN 0-7695-0299-7.
• High-Integrity System Specification and Design, by Jonathan Bowen and Mike Hinchey. Springer-Verlag, 1999. ISBN 3-540-76226-4.

See also:

• Safety-critical books listed by Amazon.com USA site.
• Safety-critical books listed by Amazon.co.uk UK site.

CompaniesEdit

The following provide safety-critical systems consultancy:

• Adelard, London, UK. Consultancy in the area of dependable systems, including safety cases. See a useful list of safety-related links, including standards.
• Aonix, San Francisco, California, USA. See safety-critical solutions using Ada.
• Critical Software Lda, Coimbra, Portugal. See links.
• CSE International Ltd (The Centre for Software Engineering), Flixborough, UK. Consultancy in computer-based high integrity and safety-critical systems and their software. Chairman and Managing Director: Phil Bennett.
• ERA Technology, UK. System safety engineering services, independent safety consultancy, safety design expertise for electronics and programmable electronic systems.
• Escher Technologies, UK. Consultancy and formal verification for critical software written in C/C++ subsets or generated from specifications. See ArC and Perfect Developer.
• MSL, UK. Engineering Integrity Systems.
• Praxis Critical Systems, Bath, UK. Specialises in all aspects of high-integrity systems. See SPARK, an integrated approach to the design, development and verification of complex software.
• Programming Research Ltd, Surrey, UK. Software quality. See QA C and the MISRA Standard. See also MISRA.
• Redifon MEL, UK. See Air Traffic Control.
• Safeware Engineering Corporation, Everett, WA, USA. Chief Executive Officer: Nancy G. Leveson.
• Savive Pty Ltd, Australia. Consulting and training services.
• TekSci, Seattle, WA, USA.
• Validated Software Corporation, Half Moon Bay, CA, USA.

MeetingsEdit

• Australian Workshop on Safety Critical Systems and Software, organized by the Australian Computer Society's Technical Committee on Safety Critical Systems.
• COMPASS (Annual IEEE Conference on COMPuter ASSurance).
• CSR events calendar, Centre for Software Reliability, UK.
• DCCA-7 (7th IFIP International Working Conference on Dependable Computing for Critical Applications) San Jose, California, USA, 6-8 January 1999.
• ESREL'98 (European Safety and Reliability Conference) Trondheim, Norway, 17–19 June 1998. Supported by ESRA.
• EWICS events.
• High Assurance Systems Engineering Symposium (HASE). IEEE International Workshop on Safety-Critical Electronic Components and Systems, Austin, Texas, USA, 29-31 August 2001.
• International System Safety Conference.
• PRFTS (Pacific Rim International Symposium on Fault-Tolerant Systems).
• SAFECOMP (Annual IFAC Symposium on SAFEty of COMPuter Control Systems). See past meetings:
  • SAFECOMP'89 proceedings.
  • SAFECOMP'90 proceedings.
  • SAFECOMP'91 proceedings.
  • SAFECOMP'92 proceedings.
  • SAFECOMP'93 proceedings.
  • SAFECOMP'94, the 13th International Conference on Computer Safety, Reliability and Security, Anaheim, California, USA, 23-26 October 1994. No known online information available.
  • SAFECOMP'95, the 14th International Conference on Computer Safety, Reliability and Security, Villa Carlotta, Belgirate, Italy, 11–13 October 1995.
  • SAFECOMP'96, the 15th international Conference on Computer Safety, Reliability and Security, Vienna, Austria, 23–25 October 1996.
  • SAFECOMP'97, York, UK, 8–10 September 1997.
  • SAFECOMP'98, Heidelberg, Germany, 5–7 October 1998.
  • SAFECOMP'99, Toulouse, France, 27–29 September 1999.
  • SAFECOMP 2000, Rotterdam, The Netherlands, 24–27 October 2000.
  • SAFECOMP 2001, Budapest, Hungary, 26–28 September 2001.
• SAFEPROCESS'91 proceedings.
• SSS (Safety-critical Systems Symposium). Annual UK Safety-Critical Systems Club symposium. Contact Joan.Atkinson@newcastle.ac.uk for further information.
• 2nd Workshop on Human Error, Safety, and System Development, Seattle, Washington, USA, 1–2 April 1998.

---

This page was mentioned in:

Safety-critical systems and the World Wide Web, by Jonathan Bowen. Safety Systems: The Safety-Critical Systems Club Newsletter, 4(2):16, January 1995.

Final thought

"If the automobile had followed the same development cycle as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year, killing everyone inside."

— Robert X. Cringely

---

Last updated by Jonathan Bowen, 1 April 2009.
Further information for possible inclusion is welcome.