AppAssure Replay Review

A complete overview of Replay and its features can be found on their website: http://www.appassure.com/solutions/backup/

I have been using this product for about 1.5 years and was drawn to it because of the impressive feature set and very competitive price. My opinions with this product have varied greatly in the time I have used it. I have loved it and I have hated it. Prior to this product I have used DoubleTake, Acronis, BackupExec and EMCs product Replistor. I have over 10 years of network administration experience and have used countless products in this arena so I have high standards :)

My requirements for this new backup/dr product were as follows: 

• Offsite Replication
• Quick and reliable backups
• The ability to recover entire VMs
• Mailbox level Exchange and SQL backups
• Affordable

One of the Replay sales folks got a hold of me at the right time - as I was looking to replace my expensive and cumbersome DR product. They gave a great demo and I was sold.

I started on version 4.3 or 4.4 - I can’t remember. Currently, they are on 4.6. I really want to love this product, but the reality is that I have been burned by it as many times as it has saved my butt. Here is list of the technical issues I've faced over the last 1.5 years:

• After installing the Replay agent on one box, it now takes 15 minutes to boot up. This is confirmed by safe mode stalling on one of the Replay drivers.
• A different box consistently blue screened every time the agent was installed. I worked for hours with support to try and resolve this but never got a resolution. We rebuilt the box and that is how it got fixed.
• Replication would often get corrupted which mean having to re-seed the drive. This is very time consuming considering you have to copy a lot of data to a USB drive and then FedEx it to your DR site.
• If you want to archive a set of backups by adding a new drive and keeping the old drive... there is no simple way to get back to those archived "recovery points." You have to go through a series of registry hacks to get to them. ARG.
• There is no replication throttling. This means the product will consume all your bandwidth when it is trying to replicate. All of their competitor products have this feature.
• On a recent server recovery, I successfully restored a server but when the server booted I was presented with the "black screen of death." Of course this happened at 11PM and I didn’t get a solution until 3AM. Long day. I ended up using a different recovery point which fixed the issue.
• Replay does not handle big servers very well. I was trying to use it with a 400GB Exchange Server and encountered numerous issues. When the system does its "online roll-ups" the recovery points are totally unavailable which means you cant run restores during this process. This would be a major problem if you were in the situation where you needed to recover quickly. The product should be able to do all its own maintenance in the background so it doesn't disrupt your environment!
• The console is very slow and crashes. This has been improved in the latest version but it is still buggy.
• The "console/core" isn’t very compatible with older Replay agents. Meaning, if you upgrade your console/core, you almost always have to update the agents. Updates usually require a reboot.

I will say that the product has consistently gotten better over time but as soon as I start to get excited about the product, I am usually quickly reminded why this is a love hate relationship. 

This company has potential but they have areas that need major improvement:

• Their support is just bad but getting better. Level 1 folks have managed to help me about 10% of the time and it is mostly just running through troubleshooting steps I have already performed. I often have to escalate to get real answers. In fact, on one of my issues I went all the way to the CEO before I got a response. Nothing personal to any of these folks, they are all very nice and probably understaffed.
• Their support is slow to respond. More often than not, I have to follow up with them. Everyone should model their support after Microsoft. The sense of ownership and follow up by MS engineers is tremendous.
• They have spotty 24 hour support. Good luck trying to get them on their "off hours." As a DR product you would think there would be good 24 hour support.
• Their account support is horrible. It took me 2 months to get a simple answer to a problem with a renewal I had. I sent 15 emails, called and then finally contacted support to have them walk over and get my account person to answer my question. If the product hadn’t been so cheap, I would have dropped them right then and there.
• Their online knowledge base is horrible. I usually try to hit this before I open a ticket but I think I found 1 solution, the rest had to be called in which = more time.

That is a lot of negative; let me speak to the positives about the product:

• If you have a smaller Exchange environment, say less than 100GB the Exchange piece is very nice. You can recover emails quickly and usually pain free.
• They have very competitive pricing to get you in the door.
• You can export your recovery points to a stand by Virtual Machine. This is incredibly cool and useful. The problem here is that you're obviously limited by your network throughput. If you're trying to export a 200GB server to a VM, be prepared to wait.
• The replication seems to work nicely in the newest version. If you have plenty of bandwidth the lack of throttling probably doesn't bother you. 
• The compression and de-duplication are fantastic. Bravo!
• Your recovery points can be validated, which means the software simulates a mount with the backup to make sure its good. This is a fantastic feature. You get a green check mark to indicate the backup is good. 
• In the latest release they seem to have fixed the slow console.

Regardless of my heartache with this product I think I will continue to use it. They'll get one more year of me! Since the license renewals are based on list price, they can get expensive but with all the work I've put into this product I need to see if they can continue to improve it. I've never been this patient with a product but I think Replay has a lot of potential; they're just not there yet. They need much better support and need to hire more developers to crank out the bug fixes and feature sets that Replay needs.

****UPDATE**** 11-9-2011
I must say, AppAssure has certainly been attentive to my bitching. They've reached out to me a number of times to resolve and ease my pain. I've personally spoken to their CEO and I am enthused with the future of the product. It sounds like some very cool things are coming out in the 5.0 release. I was also impressed with their dedication to fixing older issues and stability problems. I've noticed more and more stability as we're now running 4.7. They're moving in the right direction...

Offensive Security Certified Expert

I will start this post the same way I started my post on the OSCP certification, with a slight modification:

"This was one of the hardest THE HARDEST thing I have ever done in my life both academically and professionally. This course is not for the faint of heart and requires a lot of self discipline, perseverance and a very understanding wife."

The Offensive Security guys recommend taking the "Pentesting with Backtrack" course and successfully completing the OSCP exam challenge before you take the "Cracking the Perimeter" course. After the CTP class, you can take your Offensive Security Certified Expert exam challenge and if you pass, you become an OSCE. The OSCE course and exam challenge are significantly harder than the OSCP.

The OSCE is very different from the OSCP and I never thought I would even attempt the OSCE after the pain I endured from the OSCP. To take the Cracking the Perimeter course you have to pass an initial challenge before they will even take your money to sign up -->  http://www.fc4.me/

You have to obtain the 16 byte registration key -- sounds simple enough, eh? This is their attempt to weed out the weak! I attempted this challenge one evening, just to see if I could do it. I managed to get the registration key and submit the registration form but now I had a real predicament on my hands. "Do I let the Offensive Security guys torture me again?" The answer was clearly YES, I need more pain.

So the journey begins.

There are some notable differences between this course and the OSCP course:

1. The lab for the OSCE is not stocked full of vulnerable systems to compromise. In fact, its only a handful of boxes that you use to facilitate the course modules. Based on this, I would say you probably do not need the 60 days like I signed up for. That assumes you can dedicate 30 straight days on the material.
2. They don't need an elaborate lab for this course because a lot of the material is on exploit development. Meaning, you can hit exploit-db and practice practice practice on your own VMs.
3. In this course, you will live inside a debugger. You will become so comfortable with HEX and assembly that you will begin dreaming about EB 06. OSCP was about 5% in a debugger, OSCE is about 90%.

65 66 66 20 79 6f 75 20 68 65 78 61 64 65 63 69 6d 61 6c 20 69 20 68 61 74 65 20 79 6f 75!

The material is very interesting and for the most part, still relevant today. There was one module on Anti Virus evasion that is a little dated, however this spawned additional research and I ended up finding a way to make Metasploit payloads 100% undetectable. That is a Metasploit bind shell :) I slightly expanded work that Scriptjunkie did on this subject. This is an example of how the Offensive Security guys opened up my eyes from the course and gave me ideas so I knew what to look for.

The videos and course lab guide are brilliantly put together, just like OSCP. Here is the process I used to learn the material:

1. I watched all the videos and walked through each exercise in the lab as Muts narrated. Then, I went back and re-did everything on my own.
2. After I completed the course modules I jumped on exploit-db and started recreating all of the buffer overflow exploits I could find. I would take one, strip out everything in the middle and try to get the same results. I probably recreated 50 exploits. The point of this was to get very familiar inside a debugger and to see first hand some of the obstacles you encounter when writing exploits.
3. I would revisit the videos and course lab guide as needed.

After my 60 days of lab time it was time to take the exam. I felt like I was ready. After all, I kicked the shit out of the OSCP exam so I was feeling pretty confident about this.

Well, I wasn't ready..... at all. I failed the first exam. I only had 1/3 of the points I needed to pass. The exam is very hard but not impossible. This was the first time I have failed at something in a long time and it was a serious ego check. Not to mention I worked 17 straight hours the first day and another 15 the second day. Part of me wanted to throw the towel in because I had already learned much more than I ever thought possible and I wondered if the cert was really worth it. I thought I had reached my technical limit. That thought didn't last too long. I continued to perfect my skills and took the exam again about 3 weeks later. This time I was ready and passed. What an incredible feeling.

While I was practicing the exploitation techniques they taught me and trying to expand my skills, I managed to find a few software bugs on my own. Most of them are boring DoS, but one was a remote code execution buffer overflow.

I'm not sure I can recommend this course to everyone, it's pretty gnarly but again, brilliantly put together by Offensive Security. They certainly give you the tools to help you succeed, but as usual, they don't tell you everything you need to know. The content in this course is fascinating and if you're a security junkie you will find it thoroughly entertaining. It's too bad this cert doesn't get more notoriety because I have a much better grasp on more things security, much more than I did with OSCP. Two times now the Offensive Security folks have expanded what I thought was possible and it has really helped me in so many areas.

There is so much information to know in the infosec industry and this process taught me something important. To excel at the fastest pace possible in infosec I think you need to be on the edge of going crazy. What I mean is that there is too much to know and the only way to continue learning at an accelerated pace is to be on the edge of too much information. This is a fine line and if you can learn to balance it with your home/family life, you're in good shape, otherwise you'll go nuts.

Thanks again to offsec for making me a little more crazy and at the same time opening my eyes up to the significant issues infosec faces. At least I have a little better idea how to secure my networks and what to watch out for.

-C