Monday, May 16, 2011

AppAssure Replay Review

A complete overview of Replay and its features can be found on their website: http://www.appassure.com/solutions/backup/

I have been using this product for about 1.5 years and was drawn to it because of the impressive feature set and very competitive price. My opinions with this product have varied greatly in the time I have used it. I have loved it and I have hated it. Prior to this product I have used DoubleTake, Acronis, BackupExec and EMCs product Replistor. I have over 10 years of network administration experience and have used countless products in this arena so I have high standards :)

My requirements for this new backup/dr product were as follows:
 
  • Offsite Replication
  • Quick and reliable backups
  • The ability to recover entire VMs
  • Mailbox level Exchange and SQL backups
  • Affordable
One of the Replay sales folks got a hold of me at the right time - as I was looking to replace my expensive and cumbersome DR product. They gave a great demo and I was sold.

I started on version 4.3 or 4.4 - I can’t remember. Currently, they are on 4.6. I really want to love this product, but the reality is that I have been burned by it as many times as it has saved my butt. Here is list of the technical issues I've faced over the last 1.5 years:
  • After installing the Replay agent on one box, it now takes 15 minutes to boot up. This is confirmed by safe mode stalling on one of the Replay drivers.
  • A different box consistently blue screened every time the agent was installed. I worked for hours with support to try and resolve this but never got a resolution. We rebuilt the box and that is how it got fixed.
  • Replication would often get corrupted which mean having to re-seed the drive. This is very time consuming considering you have to copy a lot of data to a USB drive and then FedEx it to your DR site.
  • If you want to archive a set of backups by adding a new drive and keeping the old drive... there is no simple way to get back to those archived "recovery points." You have to go through a series of registry hacks to get to them. ARG.
  • There is no replication throttling. This means the product will consume all your bandwidth when it is trying to replicate. All of their competitor products have this feature.
  • On a recent server recovery, I successfully restored a server but when the server booted I was presented with the "black screen of death." Of course this happened at 11PM and I didn’t get a solution until 3AM. Long day. I ended up using a different recovery point which fixed the issue.
  • Replay does not handle big servers very well. I was trying to use it with a 400GB Exchange Server and encountered numerous issues. When the system does its "online roll-ups" the recovery points are totally unavailable which means you cant run restores during this process. This would be a major problem if you were in the situation where you needed to recover quickly. The product should be able to do all its own maintenance in the background so it doesn't disrupt your environment!
  • The console is very slow and crashes. This has been improved in the latest version but it is still buggy.
  • The "console/core" isn’t very compatible with older Replay agents. Meaning, if you upgrade your console/core, you almost always have to update the agents. Updates usually require a reboot.
I will say that the product has consistently gotten better over time but as soon as I start to get excited about the product, I am usually quickly reminded why this is a love hate relationship. 
This company has potential but they have areas that need major improvement:
  • Their support is just bad but getting better. Level 1 folks have managed to help me about 10% of the time and it is mostly just running through troubleshooting steps I have already performed. I often have to escalate to get real answers. In fact, on one of my issues I went all the way to the CEO before I got a response. Nothing personal to any of these folks, they are all very nice and probably understaffed.
  • Their support is slow to respond. More often than not, I have to follow up with them. Everyone should model their support after Microsoft. The sense of ownership and follow up by MS engineers is tremendous.
  • They have spotty 24 hour support. Good luck trying to get them on their "off hours." As a DR product you would think there would be good 24 hour support.
  • Their account support is horrible. It took me 2 months to get a simple answer to a problem with a renewal I had. I sent 15 emails, called and then finally contacted support to have them walk over and get my account person to answer my question. If the product hadn’t been so cheap, I would have dropped them right then and there.
  • Their online knowledge base is horrible. I usually try to hit this before I open a ticket but I think I found 1 solution, the rest had to be called in which = more time.
That is a lot of negative; let me speak to the positives about the product:
  • If you have a smaller Exchange environment, say less than 100GB the Exchange piece is very nice. You can recover emails quickly and usually pain free.
  • They have very competitive pricing to get you in the door.
  • You can export your recovery points to a stand by Virtual Machine. This is incredibly cool and useful. The problem here is that you're obviously limited by your network throughput. If you're trying to export a 200GB server to a VM, be prepared to wait.
  • The replication seems to work nicely in the newest version. If you have plenty of bandwidth the lack of throttling probably doesn't bother you. 
  • The compression and de-duplication are fantastic. Bravo!
  • Your recovery points can be validated, which means the software simulates a mount with the backup to make sure its good. This is a fantastic feature. You get a green check mark to indicate the backup is good. 
  • In the latest release they seem to have fixed the slow console.
Regardless of my heartache with this product I think I will continue to use it. They'll get one more year of me! Since the license renewals are based on list price, they can get expensive but with all the work I've put into this product I need to see if they can continue to improve it. I've never been this patient with a product but I think Replay has a lot of potential; they're just not there yet. They need much better support and need to hire more developers to crank out the bug fixes and feature sets that Replay needs.

****UPDATE**** 11-9-2011
I must say, AppAssure has certainly been attentive to my bitching. They've reached out to me a number of times to resolve and ease my pain. I've personally spoken to their CEO and I am enthused with the future of the product. It sounds like some very cool things are coming out in the 5.0 release. I was also impressed with their dedication to fixing older issues and stability problems. I've noticed more and more stability as we're now running 4.7. They're moving in the right direction...

33 comments:

support@ontheready.com said...

Hi and thanks for the review, have you ever looked into Axcient? I am trying to compare both products.

Craig said...

No I haven't, sorry! Let me know if you end up going with it.

Blunderdog said...

You mention 400GB Exchange server. Do you mean one database. We have 14 databases ranging from 35GB to 220GB. How would Appasure work in this environment?

Blunderdog said...

Second comment to be sure I get responses in gmail.

Craig said...

Blunderdog - I had 4 databases at about 100GB each. Your exchange environment is much larger and you'll likely see the issues I faced, but on a larger scale.

The problem that you're going to face is when it does the "rollup" function. This is when it combines/deletes data to save space. Your recovery points (backups) will be completely unavailable during this process so if you had to restore, you'd be in trouble.

There may also be an issue putting this all onto one "core" (backup server), because it has to go through and verify all the recovery points at night before it will truncate your exchange logs. I imagine there isn't enough time in a day to do this with the size of Exchange environment you have...

There may be a better strategy for that size of environment with Replay but I am not certain. However, If you do get a demo from them, make sure to ask them about these points. I didn't know enough about the product at the time to ask them and had to find out on my own.

Anonymous said...

I began using Replay in 2008, after upgrading to Exchange 2007. They were very helpful when I set up an off-line VM recovery system and I always received quick response to my questions. Fast forward to 2011. I was away for 2 years due to medical issues and during my absence the DR system was "redeployed". On my return I called Appasure and spoke to a salesperson who offered to get me relicensed on v2 and send me info so I could purchase a current license. Two days after that I received a curt email from Laureen Sander (no title or job description) telling me they will not assist me until I pay $2360.00 for a support agreement. Aargg! Tell me that up front. Do not waste my time stringing me along then sending a curt "pay up or shut up" message. Worst customer service ever! Appasure lost a customer over that one.

thegadget said...

So, Craig, I am also looking into Appassure. Thank you for such a in depth review on the product. Of course the sales folks will never tell you the weaknesses. He did mention that you will have to pay all the unpaid support agreements to continue support no matter how many years later. Too bad, escpecially with school district budget cuts.

I don't have an exchange server thankfully. We have moved to the cloud (thank the Lord!). But we do have several SQL servers in the district. Did you experience any issues with SQL 2005 or 2008? Did you use the product on 2008 R2 HyperV server?

Thank you again for all your information!

Craig said...

Gadget - I've actually had good results with the SQL piece. Make sure you read the admin guide on the SQL portion because there are certain things you have to do to get the "verification check mark" that indicates a good backup.

You can get all those docs here: http://www.appassure.com/resources/technical-documentation/

Anonymous said...

We signed on as a reseller and sold Replay to two clients. We had proposals on the table for two others but started encountering problems right away. We found the support to be the worst we've ever encountered. Additionally, every patch seems to break something and the fix is always in the next patch. Further, there's no notification of fixes and patches, you need to find out about them by calling support, which they seem to discourage by treating you like crap. We bought the lie about keeping virtual hot-spare machines ready to spin up. Guess what... it doesn't work with ESXi. Early on I dug a little deeper to see what else was out there and we discovered StorageCraft. We haven't sold another copy of AppAssure since and will replace the two AppAssure installs as soon as possible. We now have several clients using StorageCraft to back up to local NAS and also to our NOC. For clients with VMware Essentials (Roughly $700) we can keep a hot spare ESXi box with their mission critical servers ready to spin up at a moment's notice. It takes just a few minutes to roll up the most recent backups into the VM and finalize it to go live. It totally rocks.

Run as fast and as far as you can from AppAssure. It's been a huge disappointment for us once we discovered that far better products were available int he same price range.

Craig said...

Anonymous - You're the second person to recommend that product. I am going to have to try it out. If you come back to this blog, send me a message from the contact form on my main page so I can talk to you more in detail if you're a reseller.

Since I wrote this post about Appassure I have had to open about 10 more tickets for various issues. My patience is running extremely thin.

Jim G said...

Thanks for this great review. I too am looking at this product. My environments are a bit different. I use Hyper-V hosts to host all front-end servers as VMs. At present, I use DoubleTake to replicate the VMs from host to host for failover (no front-side replication) and a backup products (ArcServe, BackupExec, NTBackup) for backups first to disk, then to tape. I have Exchange environments that total as much as 500GB, plus SQL, MySQL and other application databases. Given that all my machines start as VMs and that I'm not replicating off-site, can this product work for me? Sounds like the large Exchange DBs will be the deal breaker.

What do you think?

Jim Graczyk

Craig said...

Jim - I have not used Replay with HyperV, only ESX.

Per your replication question, they seemed to have fixed the previous replication issues in this build (4.6.1.31257). However, I currently have a ticket open with them because one of my VMs refuses to export to my ESX host properly. They told me "Development" is looking into it which usually means I'm SOL. I think that if you're doing replication over the LAN on Gigabit connections, it would work just fine.

The huge backups will be problematic, take my experience on this one. Like I said before the biggest problem is when it begins doing its "rollups" (where it consolidates your recovery points) During this time you cannot restore data from those backups. If you have a ton of data, this can takes 2-3 days to run. In my environment that just isn't acceptable. Make sure you ask them about this if you demo, they may have a new strategy for this.

There have been a few people on here recommending "StorageCraft" - I would probably give that a demo as well. Please report back with what you go with! Good luck!

Anonymous said...

Thank for your thorough post and review of AppAssure. I too have been looking at several backup products to replace our standalone and varying backups for all our clients. We've trid Acronis, Vembu, Gillware, Symantec and other manufacturers. I continue to hear great things both online and in person from my peers at conferences about StorageCraft. I'm using their demo and very happy so far. Just my $0.02 worth.

Anonymous said...

We are currently trialing Appassure for a customer. We also have many sites using Shadow protect from storage craft. As far as I can tell Shadow protect is reliable however it does not have the three features we were after for this customer.
1 De-duplication
2 Replication (they have multiple sites and they want to replicate their critical servers over the wan)
3 Virtual standbys that are updated continuously.

my 2 cents

Craig said...

I just had a StorageProtect guy in to demo the product. They currently do have replication but it sounds kind of clunky. I believe they use an FTP type of transfer. The rep told me that they are completely re-vamping the replication piece that is supposed to be out in their next major release, supposedly at the end of 2011. I was very impressed by my initial demo of this product however, I was also impressed with AppAssure's. The one thing I have not found on StorageProtect is bad reviews. It seems most people that use it, love it. When they get the new features I am looking for I will demo the product and do a full review.

Craig said...

They also have virtual standby --- its called their "HeadStart Restore." You should read about the technology, it sounds impressive http://www.storagecraft.com/blog/tag/headstart-restore-recovery-time-objective-shadowprotect-backup-software-server-backup-disaster-recovery/

Anonymous said...

We purchased Replay after testing it in a lab environment and we are having the same sort of issues. Slow interface, spotty support, really bad seeding issues, interference with our clients work. It even took out a brand new server to where it was rebooting every couple hours.

I'd really like to hear more options, or more about StorageCraft from others that are using it in the real world.

Anonymous said...

Spot on with your analysis. I would saw if you are an MSP stay as far away as you can from this product. We rolled out this product to over 250 servers and it has been a headache ever since.

This product was obviously designed for in-house IT, with one location.

BRAINES said...

Has anyone evaluated Fastback from IBM , as it seems very much the same in capabilities as Appassure Replay. Except I believe it does not have the physical server export to VMDK to a virtual standby server. IBM does live demos every other week. Here is the schedule if you just want to watch and ask questions. http://www-01.ibm.com/software/tivoli/products/fastback/live-demos.html

Goose said...

I have the same pain with this application as others, using it to replicate and stand up to a DR site, AD, Exchange, SQL, and app server.

Going to upgrade to 4.7 on the 14th, hopefully it will fix the ESXi issue, my guess is it won't even thought tech support says it will. I will be testing Storage Craft and Arc serve before the renewal comes due.

wRx7M said...

I saw replay 4 advertised in spiceworks and the recorded demo of the Exchange server live replay hooked me.

Since then I have had the live demo which was also very nice. I have just gotten around to installing the core and agent on two respective servers and am having intermittent issues with the console. So far I have run about 8 restores that have supposedly completed successfully only to check the server and find that they are identical to before the restore.

I have rebooted the core and agent servers multiple times and still can't get it to work...

I am going to checkout the storageproject and others mentioned here.

Thanks for the info, everyone.

BitterDeveloper said...

I own a cloud based hosting product, and have created an entire DR product around AA, while I won't disagree with some of the QA, etc of the product. I have not had any of these issues, we have about 500+ servers under protection ranging from the small VM range of 20-30GB up to very large DAG based Exchange server > 1TB in size. The roll-ups also are processor dependent. You're talking compression and dedupe of a live image chain. That is no small amount of CPU, especially with 69 other servers still doing backups up at the same time. It also depends on the I/O that its being backed up to, our appliance is basically a small SAN with 12 drives, to get the spindles to generate the I/O needed to do your backups, including the rollups, VM hot spare, etc.

Of those 500 servers, about 300 are on a second appliance due to the customers having a DR facility, either a DC location, or a second office or campus building. The other 200 are in our cloud always replicated. We are large enough that maybe I get priority support, but while I agree with the delay and 24 hour piece, I would grade them a 6.5 / 10 as far as overall help desk support. I do agree that until we got to a certain size the internal operational support, licensing, etc was more like 2 / 10. There was a lot of turnover, and I think the people just didn't know anything.

I highly recommend 4.71, and ESXi is supported, but you need a VC to get the API. That is a VMWare not an Appassure issue. Realize that ESXi free is probably dead, with 5.0 you only get 8GB allocation so its worthless except for a few small VM test hypervisor. But essentials at ~ $500 is a cheap option to get a nice DR cluster, we use that for our 2 appliance + 1 ESXi box...

Storagecraft isn't too bad, but I think AA is a much better cleaner concise product. I'm not sure why the replication issues have been what you've experiencing unless again its I/O based. Not saying I don't get the occasional replication issue, but we also make sure the bandwidth is there to keep a 2 day swag for any issues. We use a network monitoring solution to help augment the features as my issues are more about reporting, multi-tenant options, etc

We're beta'ing the 5.0 product and its another nice step although not fully baked, its web based, etc so it should remove the console issue.

Larry said...

We have been using Storagecraft for years and have recently launched an MSP based DR service based around their new replication / headstart restore (hyper-v & vmware) - it's a great product set (mailbox level restore is not built in but they have GRE as an add-on which is essentially ontrack power control which is another excellent product.) AppAssure was recommended to me & am going to look at it but am concerned that datto dropped it in favour of storagecraft for their appliance

CPUGEEK said...

-You guys hit the nail right on the head with your observations. Replay works well when it works, but it's a nightmare when it doesn't. -Support is good when you get them. -Replay breaks way too often. -Large servers are a beast to manage, and reseeding/resynching/redoing full base images/rereplicating takes forever and potentially consumes the whole core. They desperately need a better way to centrally manage for MSPs and to do a central one-click update without reboots. If I am not mistaken, the last push didn't require reboots.


Overall it's still a phenominal product when it works. I can't wait to see 5.0.

No name said...

I am using this product for an year and really consider look for other solution. Their product seems fine but the support is horrible.
Their tech support sucks. When you call for help, message taker will take all the information and tell you "A engineer will contact you as ther are available." Usually they call you 2-3 days later. If you open a ticket thru online message taker send you email and asking you exactly same information no matter how detail you describe in the ticket.
Also they always blame MS for the problem. When a enginner can't figure it out the solution, they ask contact MS for the solution most of the time. They are not willing to work with you pin point it the source of the problem. If you can figure it out if it is MS problem, you would not call AppAssure from the beginning.

Read forum in their site. Their own customer complaints the support.

wRx7M said...

I am revisiting appassure since dell bought them and have seen that there is some information about version 5 on their site but I am not sure if it has been released or still in beta. Anyone know if it has been released yet?

Craig said...

Looks like it has, here are the release notes: http://docs.appassure.com/display/AA50D/AppAssure+5+Release+Notes

Thanks for the note on Dell, I had not heard that. Wonder if that will help or hurt?

Ed from Ohio said...

We have AppAssure Replay on 2 hosts at 2 locations backing up about 70 ESXi VM's and a few physical boxes.

I'm not happy at all with the product. If it has an issue with a "snapshot" (their term for backup), it stops trying to back it up in the future so you have to manually do it. It requires an agent on every single machine it has to back up. Rollups are slow and you can't tell what it's doing. You just get an "Abort Rollup" button. Their support is absolutely horrible - it takes a solid week for a real support person to call you back. Their agent has crashed multiple VM's multiple times. It can't backup Linux boxes.

The sales rep said you can restore the shell of a VM and "stand it up" while it continues to restore data in the background. That was a flat-out lie. The list goes on and on.

With all those problems, you'd think the product was cheap. It is horribly expensive. I really wish our company had picked something like Veeam instead.

OBFF said...

Ed, you hit the nail on the head. We trialed it on a limited basis, were fairly impressed, and then purchased 100 licenses for use with our clients. This was a terrible, terrible mistake.

Their support is horrendous. I actually had their support guy tell me that the Exchange mountability checks were at the "block" level. How the hell do you check to see if individual blocks of data are mountable as an Exchange DB? Long story short, we had an Exchange server crash and our fully verified Replay recovery point didn't work. In addition to that, the recovery process for 300GB of data took almost 24 hours. We had to go through multiple restores trying different things, so the server was down for a solid week. In the end we ended up having to completely rebuild the server and manually copy the DB files to it.

Their export to VM function did not work at all and was much more trouble than it's worth when I have been able to create a VM from it. Overall a complete and total waste of tens of thousands of dollars.

Other issues:

Replication breaks frequently.

They are constantly releasing new versions. If you are using replication, if you update 1 core you have to update all cores, because the replication is not compatible between core versions.

Updating an agent requires a reboot.

The application is incredibly resource intensive, especially on memory. Doing a restore of any size requires a server with at least 24GB of RAM.

The application does not backup the system state. If you get a corrupt AD on an SBS box (has happened to us a few times over the years with various customers), you're restoring the entire volume.

The actual bare-metal restore process is a huge pain. You have to create a specialized disk for the hardware you're using, boot the server off of it and then point it at a core. Your server was the core? Guess what? You have to get another computer, install the core version you had on there, modify the registry to point at the Replay repository and then wait while it streams over a network connection.

I would steer well clear of this product. It will cost you much more than just the sticker price.

Unknown said...

We also purchased 1 license to backup our Exchange 2010 mailbox server. While it worked OK during the trial, it has been a nightmare ever since. Support is absolutly horrible! The sales people sold us on the lie of how quick you could restore in case of a corrupt database. Our average database is 200GB and there is no way this won't take several hours.
Integrity checks take days to run, if they ever do run and when they don't your tranaction logs don't get truncated. I will probably go back to Arcserver and do tape backups until I can find something that works!

Daniel Messenger said...

I just ran across this review as I have been researching AppAssure since I am very unhappy with Symantec BE and am looking at other solutions. I have a very small network but I'm looking at these options: continuing BE, Unitrends appliance, Symantec appliance, and AppAssure. I really like what I've seen from Unitrends and think that is the best option, but so far haven't found a review like this one. Anyone used Unitrends?

Chrispy said...

I have used Replay for 1.5years now.
There is always some undocumented bug that I come buy. I feel like I am their QA. As far as support, you always get conflicting answers from the level 1 and 2 techs. It got so much worst with the Dell purchase, and AppAssure 5 does not support 4 at all. The migration toll stinks btw, and good luck installing on server 2003 without the OS installation disk.

Chrispy said...

I have been using them since 4.7 and before Dell purchased them. I'll be frank, they support really sucks now. Hold times are over 15 minutes. You have to call them all the time cause they do not follow up and close your case! Dell ruined them and made a good solution into a crappy one. Thanks Dell! Avoid this at all cost and use someone else!