Shop Mobile More Submit  Join Login

How We Debug in DT

Thu Sep 13, 2012, 12:06 PM by fartprincess:iconfartprincess:
As developers, we often have to debug complex data across several different layers: server-side PHP, Javascript, load time, server information, memcache events, how pages are routed, along with several other pieces of data.

In dt, we use an internal system called DRE (Developer Runtime Environment), which allows us to grab information about how the code we write is executed while making a minimal impact on the code itself.

What makes DRE especially awesome?

We can use it to help debug and solve problems that other users are having. To accomplish this, once we are aware that a particular user is having an issue, we can add DRE logging to this user, usually limiting the logging to the part of the site in which they are having the problem. We can then use the log to check for any errors that might show up.

For example, say a user named testera is having trouble with a journal. We can log debug data for him when he visits journal pages. Examining that log, several pieces of information are stored, many of which will give us a better idea of where the point of failure is. We might want to check privileges the user has or what roles they have in a group if that's where the journal is being posted.

We take this type of logging very seriously, so don't be :paranoid:, because we only use it as a last resort, typically only using it within our own personal development spaces with test data.

We also store the logs so we're able to link them to other developers (and even then, we only store those logs for a few hours).

Why is it helpful to developers?

Because we load all debug information in a single window, it is accessible either by clicking a box at the top of any page or through F2. DRE can log simple data types but it can also provide a highly readable breakdown of complex objects using nested tables.

We also use it to set up alerts. If a high-risk condition occurs on a particular page, the box used to display DRE will turn red and pulsate to make it more apparent to us that there is a problem. This could be something as simple as a failure to retrieve data or something as severe as a failure to include a file (or worse, a fatal error).

More importantly, it allows us to trace data and debug across multiple layers, including processes that are non-web-facing or for asynchronous tasks, where the data may not be immediately available at runtime.

It's not just for fixing bugs; it's also for helping us spotting slow-loading modules and queries and seeing where we can better optimize the number of queries we do need to run on any given page.

For example, on this page, we have 12 separate queries that run:

This is pretty good, given all the data that has to be loaded to make DeviantArt run.

But on a different page, we might have 52 queries, some which are very similar and loading data from the same tables. If we can find a way to cull them together, we can reduce the number of queries, which will help the site perform better. We have a way of handling this in dt, a technique called wenching (which is combining several queries into a single query--we'll get to this in a future article), but we might not be aware that something needs to be wenched until we get a composite look of everything taking place on the page. This is where DRE comes into play.

We can also use it for profiling to see when a file in particular is taking a long time to load. This might be because the code in that file has inefficiencies that need to be refactored. For example, here's a page just to show you how this might look:

Eek. Bad, right? Fortunately, this isn't something we see too much of. And when we do, we try to remedy it.

So how does it work?

This is a silly impractical example, but it'll do.

$orders = $this->get_orders();
dre_trace('orders', $orders);

If get_orders() returns an object of orders, DRE will trace that object into a table, labeled "orders". Simple enough.

"This sounds like it could be kind of messy when you have a bunch of developers all using it! If everyone is using DRE, the panel would be completely overwhelmed with junk I don't care about!"

Yep. That's why we use dre_trace() sparingly. We have several functions for DRE, one which takes a parameter to filter the message to only those who want to see it. For example, if I wanted to make sure only I saw $orders from the aforementioned example in our DRE panel:

$orders = $this->get_orders();
dre_cond_trace('fartprincess', 'orders', $orders);

At the end of the day, this utility saves developers a massive amount of time.

Add a Comment:
20after4 Featured By Owner Oct 9, 2013  Hobbyist Photographer
For those of us who don't have the privilege of using the cool %dt tools, there is an open source debugging tool that has some similarities to dre. Check out, it's pretty cool, I don't miss DRE now that I've discovered debugbar.
banks Featured By Owner Oct 10, 2013
Cool, that looks really nice Mukunda.
Itti Featured By Owner Oct 5, 2012  Hobbyist General Artist
Nice! I always thought that if you get a rare, non-reproducable error, then there wasn't much you could do about it. It's really interesting to see that you have this option available for some of those particular "special case" bugs.

Keep up the interesting articles! :D
admx Featured By Owner Sep 18, 2012
I have no idea what this is but I love it :la:
OrrinFox Featured By Owner Sep 16, 2012
Well this looks like a wonderful development platform here. I would definitely like the idea of working with something like this especially when it comes to something like server side coding. definitely a lot more complex than the simple development tools in chrome :lmao:
Anyways looks like wonderful work here none the less to say.

P.s. Now i know this is something that devs arent "allowed to speak of" about future plans or anything.. but just in general when is any development going to be done on dAmn? no specific date but I was wondering what the overall status on that was going to be.
SaTaNiA Featured By Owner Sep 14, 2012  Hobbyist Digital Artist
Interresting read even if I don't understand everything :) :la:
zeruch Featured By Owner Sep 14, 2012  Professional General Artist
How much actual latency does this add, and is the tool fully homegrown or use any of the new libs/gems out there?
kemayo Featured By Owner Sep 14, 2012
For normal users it adds almost nothing; no-op versions of the functions are used. For developers with it enabled it can noticeably impact load times and, in particular, memory use. We actually run our developer environments with a higher memory limit than the site itself to compensate.

It's totally homegrown. We've also had it for quite a while; it's only had subtle changes since I joined dA over 4 years ago.
timmy64 Featured By Owner Sep 13, 2012  Hobbyist Artist
Is it sad that while I understood all that, I'm actually terrible at coding? :slow:
namenotrequired Featured By Owner Sep 14, 2012  Student Interface Designer
Just means it's well written I suppose :lmao:
rotane Featured By Owner Sep 14, 2012  Professional
Not at all, sounds just like me ;)
timmy64 Featured By Owner Sep 14, 2012  Hobbyist Artist
pestavous Featured By Owner Sep 13, 2012
one day, i may be able to actually understand what this article said. all i understood, is that under the hood of deviantART are collections of complex methods to handle the flood of data users generate.

it's really cool to learn what you're doing there to make this site work!
silverluna Featured By Owner Sep 13, 2012   General Artist
Interesting to know about the logging, especially since, when I sent a bug notice, I then tend to avoid the part of the site that's 'broken' for me.
mudimba Featured By Owner Sep 13, 2012  Hobbyist
As $fartprincess said, it is a last resort. When we do resort to it, it is usually coordinated with the user (for example through helpdesk). Most of the time bug reports give us enough information that we are able to replicate the problem using our own accounts or accounts that we specially create on our test machines.
fartprincess Featured By Owner Sep 13, 2012  Professional Writer
I wouldn't worry too much about this :) We'll go to all lengths possible before enabling logging this way--it's very much a last resort and not something that happens very commonly at all.
ZCochrane Featured By Owner Sep 13, 2012  Student Photographer
Very interesting!
vast-ginger Featured By Owner Sep 13, 2012  Hobbyist Photographer
From a development perspective, is there a concurrent 'local' version that's synced with the dA live site, so that if anything gets borked on the local version, there's no danger to the live site? I know you said 'personal development spaces', but i'm curious. :D or is this a 'confidential secret'? lol
fartprincess Featured By Owner Sep 13, 2012  Professional Writer
By "personal development spaces," I mean that all developers run a local version of DeviantART on a virtual machine, which allows us to work on code without the fear of completely destroying the site while doing so :)

We then sync code to Git. You can read more about our VM here: [link] -- we're hoping to write more on the whole process in the near future so keep an eye out!
Itti Featured By Owner Oct 5, 2012  Hobbyist General Artist
That's really interesting. I am just starting to learn more about team coding - repositories and syncs and branching commits - so it was really interesting to find out how it works in an environment that I'm familiar with from the user end.

I'd love to see the update on that process when it comes :)
electricjonny Featured By Owner Sep 13, 2012  Hobbyist Photographer
Very interesting :nod:

While I don't exactly understand it all, it really sounds like you guys have a pretty good grasp on debugging and useful tools to help make this (massive) site run smoothly.
deviant-garde Featured By Owner Sep 13, 2012
Always interesting to read these kinds of articles from you guys. :thumbsup: Also seems to explain a thing or two I've seen making third-party userscripts for myself and for other people. :paranoid:
Add a Comment:

:icondt: More from dt

Featured in Collections

News I Support by namenotrequired


News and Journals by Lyricanna

More from DeviantArt


Submitted on
September 13, 2012
Submitted with


9 (who?)