Calling widespread bad habits in database administration "deadly" may seem extreme. However, when you consider
critical nature of most data, and just how damaging data loss or corruption can be to a corporation, "deadly" seems pretty dead-on.Although these habits are distressingly common among DBAs, they are curable with some shrewd management intervention. What follows is a list of seven habits we consider deadliest, along with some ideas on how to eliminate them.
Habit #1. THE LEAP OF FAITH: "We have faith in our backup."
Blind faith can be endearing, but not when it comes backing up a database. Backups should be trusted only as far as they have been tested and verified.
Cures: Have your DBAs verify that backup is succeeding regularly, preferably using a script that notifies them if there's an issue. Maintain a backup to your backup. DBAs should always use at least two backup methods. A common technique is to use those old-fashioned exports as a backup to online backups. Resource test recoveries as often as is practical. An early sign that your DBA team is either overworked or not prioritizing correctly is having a quarter go by without a test recovery. Test recoveries confirm that your backup strategy is on track, while allowing your team to practice recovery activities so they can handle them effectively when time comes.
Habit #2. GREAT EXPECTATIONS: "It will work way we expect it to. Let's go ahead."
Although not user friendly in traditional sense, Oracle is very power-user friendly— once you've been working with it for a while, you develop an instinct for way things "should" work. Although that instinct is often right, one of most dangerous habits any DBA can possess is an assumption that Oracle will "just work" way it should.
Cures: Inculcate a "practice, practice, practice" mentality throughout organization. DBAs need to rehearse activities in safe sandbox of a test environment that's designed to closely mimic behaviour of production system. The organization needs to allow time and money for them to do so. Pair inexperienced DBAs with senior ones whenever possible—or take them under your own wing. New DBAs tend to be fearless, but learning from someone else's experience can help instill some much needed paranoia. Review plans for everything. It's amazing how often DBAs say, "I've done that a hundred times, I don't need a plan." If they're heading into execution mode, they absolutely need a plan.
Habit #3. LAISSEZ-FAIRE ADMINISTRATION: "We don't need to monitor system. The users always let us know when something's wrong."
If you depend on users to inform DBA team that there's a problem, it may already be too late.
Cures: Install availability and performance monitoring systems so that issues are identified and resolved before they cause service-affecting failures. Avoid post-release software issues by working with developers and testers to ensure that all production-ready software is stable and high-performance.
Habit #4. THE MEMORY TEST: "We'll remember how this happened, and what we did to get things going again."
It may seem impossible that a DBA team would forget a massive procedure that took them weeks to get right, and yet it happens all time. In order to prevent recurring mistakes and take advantage of gained experience, documentation is essential.
Cures: Require that your DBAs maintain a comprehensive documentation library and activity diary, including a significant level of rationale, syntax, and workflow detail. Provide your team with groupware on your intranet so that these documents become searchable in an emergency. Enforce discipline of documentation and check it periodically. Ask your DBAs: When was this tablespace created, by whom, and with what SQL? What tasks were performed on a particular day? If they can't answer quickly, you'll know they've gone back to relying on memory.
