In the real world, there ain’t no absolute things. Words like ‘never’ are always wrong (notice the recursion?). A RavenDB database should never go corrupt. But it does. In this post, I’ll show you what you can do when you get an error like this:
System.InvalidOperationException: Could not open transactional storage
First, let’s talk about what could possibly lead to such a problem. As you probably know at this point, RavenDB uses ESENT as it’s storage engine. ESENT is a transactional storage engine that ships with every Windows version since Windows 2000 and powers things like the Active Directory, Windows Search Index or Exchange Server. When you get the above error, there’s almost always something wrong with the ESENT files on the disk.
Personally, I’ve seen 4 different scenarios (I’m sure there are more) that could bring your database into such a corrupted state:
- Dirty shutdown
Power outages, hardware failure or evil admin users could cause the RavenDB process to be killed unexpectedly. When this happens, there is a high chance that the ESENT data file on disk gets corrupted. - Moving between windows versions
If you just copy the data directory (instead of using Smuggler), you will likely end up with ESENT not being able to startup on the new machine. - Live migration of a virtual machine
Don’t believe hypervisor vendors like VMware or Citrix when they claim they can securely migrate a running virtual machine. They can’t. I’ve already crushed two databases while doing live migrations. - Upgrading to RavenDB 2.0 unstable
If you’re like me and can’t await the official realease of RavenDB 2.0, then please be aware that there could be issues when you upgrade an existing database.
I should say at this point, that RavenDB tries to automatically recover the database upon startup when it notices that it wasn’t shut down cleanly. However, that didn’t always work for me and I had to execute some manual steps to get my databases up and running.
Actually, it is quite simple – you just need to learn how to use the tool esentutl.exe which comes with Windows.
Here are a few things that you can do:
Defragmentation
This is most useful when you moved the data folder between different Windows version. Go into your data folder, open a command prompt and type:
esentutl /d Data
Soft recovery
On the disk, ESENT maintains all of the data within a single file. In addition to that, every transaction gets written to a log file. Using that log files, ESENT can re-run those operations and recreate a consistent state. Using this command, you should be able to recover from almost all damages:
esentutl /r RVN /l logs /s system
Repair
When things went really bad and both defragmentation and recovery didn’t help, you can fall back to a forced repair. Only consider this option when a recovery didn’t work. If you can, restore from a backup. If you can’t, this command can save your butt:
esentutl /p Data
Of course the best is to avoid running into these situations in the first place. However, if when you have this scaring “Could not open transactional storage” exception, it is good to know what to do!
Shouldn’t RavenDB do this on startup automatically? ESENT should be an implementation detail.
ESENT itself claims to do it automatically upon startup, but for some reason it doesn’t always work. I agree with you, it should be an implementation detail to RavenDB, but I wouldn’t want raven to hide such an important restore operation from me. Not only because it can take quite a lot of time, but also because I as an ops person want to know what was going on, how to prevent that in the future, etc.