Coming up with themes and titles for a blog post can be very challenging. You want something that will catch the eye, but not be so random or out of left field that no one catches your drift. This morning I had two themes battling inside my head for this post.
- Welcome to the Jungle Baby!
- Pigs In Space
You’re here so you already know who won. Pigs in space? Pigs don’t belong in space?!?! That’s work for dogs, monkeys, and humans, right? Well I’m sure some of you were just as aghast last week when you heard Chet aka @oraclenerd talking about developers in production.
In Chet’s post, I was quoted as playing both sides a bit. This is because I have played both roles. In the end I came out and declared that in the ideal world, developers would have no access to production instances. But since we don’t often live and work in the ideal world, I want to discuss some possible reasons developers would have access to production. I invite @oraclenerd to take up the opposing counterpoint on his blog 🙂
Top 10 Reasons You Might Find a Developer in Production
1: They Are Already There
That’s right, the previous administration decided at some point to let the developers in. Getting them out now is going to be problematic. They are probably going to say they can’t do their jobs without having access because they are used to having it. They may or may not have valid reasons. Don’t assume they are wrong. Maybe it’s because…
2: There is No Dev – Test – Prod Delineation
You will find a nice, clean, and separate ‘here is where we code, here is where we test, and here is where we have users running around’ ecosystem in most shops. However, I have visited more than a few places where you might have only one or two of these. I would argue this is much worse than having developers running amuck in production. Treat the condition, not the symptom.
3: Wearers of Multiple Hats
Smart people often fill more than just one role in an organization. They write application code, but they also help out with support and migration and upgrade scripts. They may even serve as a local DBA for their development instances. You can let one developer in without letting them all in, assuming they are there at your behest. I would hate working somewhere where my job title or label prevented me from helping the organization wherever my talents lended themselves to do so. Accidental DBAs, please raise your hand. Now take a picture, and email it to us so we can see it. I’ll just assume you are out there if I don’t get any pictures…
4: Putting Out Fires
It would be hard to argue that no one knows more about the inner-workings of an application than the person who wrote the application. It may become necessary for a DBA to go to a developer when they have exhausted all of their tricks when tracking down bugs and performance problems. Most developer will refuse to even look at a bug report without the appropriate logs and environmental information. If the problem is on-going and VERY painful, it might just be easier to give them access to the database. This assumes that:
-
1: They can be trusted with confidential information (or such information is protected via privileges)
B: They already know their way around the data dictionary v$ objects.
5: Your Security Sucks
If a developer wants something, they are smart enough to figure out how to get it. The ethical ones will generally stay away from places they are not supposed to be. But if you have a production database with weak or default passwords, a developer might just assume they’re allowed to be there. There are other ways to access a production database outside the proper channels, but I don’t want to give away too many secrets. Developer know how to use Google. Don’t tempt them by making it easy. And if you find them, warn them and patch up your holes, or it will be more than just their job on the chopping block.
6: It’s Only Kinda Prod
Reporting instances that are copies of production. Generally built to off-load the reporting overhead from the actual production instance. Built for all of the data nerds in the company. Accountants, bean counters, business analysts all need to run their ‘SELECT * FROM…’ queries somewhere, right? If you have developers tasked to building reporting interfaces or doing ETL work, you might find them here. This is a corollary to the previous ‘There is No Dev – Test – Prod Delineation’ rule. Can you trust your developers with the business critical data? If you are dealing with Health Insurance Portability and Accountability Act (HIPPA), I wouldn’t risk it.
7-10
I’m going to leave some space here at the end for others to fill in. Mostly because I’ve covered the the obvious ‘loopholes’ and I’m sure there are plenty of bright people out there that can share their own personal experiences here.
11 Comments
-rf is not bad in a blog comment. I hope you see something similar to
'; drop table comments cascade;– otherwise, ask your developers about 4 and 5 😉8. They’ve never been allowed to find out the consequences of typing in rm -rf / u01/$ORACLE_SID/tempdata.dbf as root on someone else’s machine.
-rf hurts to even read that in a blog comment!
My favourite reason is “a manager said he wanted a change and he wanted it now”, which is symptom of a problem with the deployment process. i.e. it should be just as easy to make the fix in test and then deploy it as it is to fix in prod.
It’s hard to think of a dev and test environment as a luxury, but maybe I’m just spoiled?
I think the big one is close to your “Putting out fires”. Bugs show up in live systems. Why didn’t Mrs Jones get her bill ? Where was McFood’s monthly discount ?
Those problems often needs access to live data. Sometimes you can let them off in a ProdCopy (but if it has real Prod data, the whole confidentiality/security thing still applies – so in many respects a ProdCopy is still a Prod database).
The READONLY thing may apply, but you can get into a fuzzy area on executes and what tables/columns/data dictionary views get included in the read.
I think you’re right, when the site is on fire, we have to bring in whoever it requires to put it out. But bringing in the developers to a production instance would probably the the last step. I say last, because it would be the most expensive solution. Every minute a developer is helping support, that’s more time lost on current development projects.
In a perfect world though, I would hope the test data would be good enough to be able to duplicate bugs found in Prod. If not, then maybe we don’t have good enough test environments. When was the last time you let a developer from Oracle login to your production database to work on a TAR? Wait, they don’t call them that anymore…you know what I mean!
Right now I’m advocating getting development resources brought in to help with a fire, and the developers would rather have their teeth pulled. We never even addressed how many developers out there actually WANT to work in PROD. I’m guessing most don’t want to. They’d rather build their stuff and let someone else support it.
@Karen I’m filing that one under too crazy to make up.
With regard to the whole security argument, with number 6, if there is no money/personal info/HIPPA involved, that might not be a concern. For example, in the case of a news site (massively multi-author blog) or something like stack overflow. If the authentication is federated through active directory or OpenId, why not give the devs readonly access to the network share where the DB backups are? Let them develop against production data.
Now, from the other side of a coin, I was once a dev at a company without DBAs. I wrote ETL processes. I deployed the code, and troubleshooted problems in production. I documented the crap out of everything so that netops and the account execs could deal with most of the support issues. I lead the charge to allow them to begin to take away my access to things. Sometimes its the management and not the devs that push for devs to have access to production.
I like your stack overflow example. Keywords being “readonly access” I think as DBAs we’re trained to say ‘No’, but I hope we’re also open to things that just make sense.
I guess devs are cheaper than DBAs, so they become de-facto DBAs. I think that’s very short-sighted, but again this is real-world time again. I love wearing the multiple hats, that job you had sounded like a lot of fun, even if it was frustrating to not have DBAs.
Justin, thanks so much for the feedback, welcome to my crazy little world!
Related to item 5, your security sucks: Vendor pays off person with production access so that they can “fix” data that the application is getting wrong, in hopes of hiding logical errors in their code. So is that item 7: Fraud?
Love Your Data – keep it safe.
@Karen – what @Justin said – ‘too crazy to make up’
Holy Wow!