Seamlessly Refresh OneTick Databases with Catalogic ECX

OneTick is a time series database that is primarily used for providing both historical and real-time services, including complex event processing and historical tick data analysis. OneTick is designed for the capture, storage, retrieval and analysis of data and needs to be able to handle inbound data at an extremely high-frequency (oftentimes with milli-, micro-, or nano-second granularity). Because of this, OneTick is widely used in the Finance world to improve trading performance through superior data management. OneTick users include hedge funds, proprietary trading firms, banks, brokers, technology and information providers, universities and more. Use cases include backtesting and quantitative research, high frequency trading signal generation, venue analysis, risk and portfolio analytics, and other generic time series analysis.

Dealing with this type of high-frequency data presents a problem when it comes to development and testing. If you have a need where multiple developers require a dedicated instance of the OneTick database to develop against, it becomes very difficult to present fresh data for these developers. Because the OneTick application does not support NAS for the data storage, it is not possible to share tick data across multiple environments. The workaround, which would include manually copying and mounting the drive containing the OneTick database to each individual development server, would take a considerable amount of time (depending on the size of that OneTick database), and by the time the database is available for development, the data is stale.

With Catalogic Software’s ECX solution, this process becomes simple. ECX is a virtual appliance that manages your existing storage (Pure Storage, NetApp, or IBM Spectrum Virtualize and Accelerate) and provides the ability to automate and orchestrate application consistent storage snapshots. Most ECX customers use ECX to help with SQL or Oracle use-cases like database refreshes, Test-Dev, DevOps, or even DR, but ECX is also able to perform backup and recovery of InterSystems Caché or SAP HANA databases and VMs in VMware environments.

So how does ECX simplify the OneTick pain point discussed earlier? In addition to the ability to backup and restore individual VMs, ECX can also perform what we call an “instant disk restore.” This means that using the storage snapshots that ECX automates, it can instantly mount a cloned drive from one VM to another. When ECX performs a backup job of a particular VM, it is essentially taking a snapshot of the storage volume that the VM sits on. Then, during an instant disk restore, the end user selects the drive that contains the OneTick database, and ECX clones the volume and then mounts that clone onto the destination server. To make it even better, because ECX is working with cloned storage volumes, the end user can create 2… 5… 10… 25 copies of that drive and it will not require additional local disk space, because they are considered zero-footprint copies. Typically, mounting a 1TB drive onto 5 different servers would require 5TB of local storage. This is not the case when working with ECX. The “Disk Restore” is also extremely fast, because ECX is not mounting an entirely new version of the database, it is instead simply mounting a clone from a snapshot.

One particular ECX user, who is using ECX for this OneTick use-case, is using scheduled restore jobs to automate daily refreshes of the OneTick database onto a half-dozen Test-Dev servers. As part of these scheduled restore jobs, they are also taking advantage of ECX’s “allow to overwrite and force clean-up of pending old session” feature. This eliminates any need for manual cleanup, etc. because this feature tells ECX to automatically clean up the previously mounted version of the drive (dismount the disk, delete the cloned volume from storage), and simply replace it with the most recent version of production data for testing. Now, on a daily basis, their developers arrive at the office with a fresh version of the OneTick database waiting for them.

If you or someone in your organization is experiencing the pain of providing instances of your OneTick database to your development team for testing, and want to know more about how Catalogic Software’s ECX can help, check out our website.

You can request a live demo, or even get a 30-day trial copy to try it yourself. We’ll be happy to help you get things set up.

Read More
09/30/2019 0 Comments

Our Thanks to Pure //Accelerate Attendees Who Helped Plant Over 1,000 Trees

Pure //Accelerate 2019 has concluded for another year, and as always it was a great event. We’d like to thank Pure Storage for being the gracious hosts they always are. And mostly, we’d like to thank those of you that participated in our No More Swag campaign. With your help, instead of giving away tradeshow swag, we donated $1,115 to plant trees. At a dollar a tree, that’s 1,115 trees planted. And with the matching donation from Pure Storage, that will double it. Pretty nice, planting 2,230 trees instead of getting another squeeze ball! And please, there is still time to donate at the link above.

For fun, I did a little back of the napkin style calculations to see what might have been. Since a squeeze ball purchased in bulk goes for about 65 cents, we can come up with the following.

  • By NOT spending the money on squeeze balls, we managed to prevent 1,715 squeeze balls from ending up in landfills (which is where they all end up).
  • We could have formed a line of squeeze balls 393 feet long when laid next to each other. Think about that. From one end of a football field to the other, and then back again to the 30-yard line. Sheesh.

The real “makes you go hmmmm” part of this is that Catalogic is just one vendor and this is from one show. Think how many total squeeze balls – and all the other stuff – gets handed out annually.

Now I have nothing against higher-value giveaways. Attendees at shows expect that prizes are part of the fun, and if as a vendor you want to offer a solid prize that someone will actually use and not just toss away a few days later, I say have at it. But handing out endless piles of tchotchkes, 99% of which are headed for the garbage within a few weeks, is something that we as an industry need to seriously think about.

Read More
09/23/2019 0 Comments

No more squeeze balls! Why we ditched swag for Pure //Accelerate 2019

With trade show season in full flight, an IT person’s fancy turns to… swag! You know, all that free giveaway stuff from tech vendors. Stuff with logos on it. Squeeze balls, pens, water bottles, and countless other tchotchkes. Who doesn’t love swag?

Well the Earth doesn’t love it all that much. What is swag but junk waiting to happen? Its half-life can’t be more than a few weeks. So this year for Pure //Accelerate 2019, Catalogic decided that we were going to ditch the swag.

At the same time, we saw that Pure was sponsoring a great charity idea for //Accelerate: Plant a Tree! For each dollar donated, a tree gets planted by the organization One Tree Planted. Pure is matching all donations up to $5,000. For our part, we decided to join in (you should too; please click the link and donate!).

When you visit our booth at //Accelerate (#608), instead of handing you a squeeze ball we’re going to make a $2 donation to Plant a Tree. And by answering a simple four question IT survey online, we’ll add another $3. So by visiting our booth you can actually plant up to five trees!

Image: A representative sample of trade show swag. How much of this do we really need?

A representative sample of trade show swag. How much of this do we really need?

But let’s be honest. The reason tech vendors sponsor shows – at considerable cost of dollars and staff time – is to talk to potential customers. And swag is designed to get potential customers to visit you. We’re hoping we can alter that mindset just a little, and maybe attendees will find it’s better to give than receive. And do you really need another squeeze ball?

In the larger scheme of things, it’s time for organizations to start thinking about swag. I still recall my first trade show, back in the mid-1990s when I was working for the late, great Ascend Communications. I was agog at all the free STUFF you could lay your hands on. I also had a couple of nine-year old twins at home: perfect swag age! On the last day of the show I had some free time, so I made a swag run (if you never saw internet boom era trade show swag, you haven’t seen swag). I ended up with two shopping bags full of stuff. How did I get it all home? I don’t remember.

The kids were sure excited by it all. And then in about two days, it was just a lot of junk laying around, and it all got tossed out pretty quickly. Such is the lifecycle of swag. And I think we can all come up with some better way to have fun with our kids then bringing them squeeze balls and light up yo-yos, but I understand the temptation, believe me.

This is a reasonable approximation of me at my first trade show.

For a while I was also one of those Lanyard Guys. You know, the guy who has twenty-five different show badges hanging by his desk. Every tech company has these guys. But what about lanyards?

At VMworld this year, there were some 20,000 attendees. That’s a lot of lanyards and badges. If you laid them end-to-end, they’d reach from Pittsburgh to São Paulo. Ok, I just made that up. But it’s still a lot of lanyards. Can you recycle them? Would it make sense to have bins out on the last day of the event, so people leaving the show can toss their lanyards into the recycling bin?

I’m not trying to bash trade shows. I think they’re great events. With all the information available on the internet, there’s still nothing as effective as having a direct discussion in front of the computer screen and helping someone figure out their IT challenges. Loads of learning happens at them; people get technical certifications. Shows are a vital part of the tech ecosystem for both vendors and users.

But when it comes to swag and waste, we can do better. In our own little way this year, we’re trying something new. If you’re attending //Accelerate, please drop by and show your support.

Read More
09/06/2019 0 Comments

Microsoft SQL Server Cluster Backup and Restore using Pure Storage and Catalogic ECX

Catalogic ECX offers a unique, effective yet user-friendly solution to protect and use your valuable assets in Microsoft SQL Server™ databases. With ECX, you can help ensure your business continues and mitigate business impact under various types of unexpected disasters. And you can also quickly deliver working copies of SQL databases for business use cases such as test-dev, DevOps, reporting, etc.

ECX automates SQL Server-aware snapshot copies on enterprise-level storage arrays including Pure Storage FlashArray, NetApp ONTAP based storage, and multiple storage systems from IBM Storage. You can manually create a snapshot whenever you want, or schedule and automate backup jobs with ECX’s Service Level Agreement (SLA) policies you define.

ECX coordinates the copy process between the application and operating system layers and the underlying storage platform to create database-aware snapshots.

You can choose to restore any snapshot of a desired date and time, even re-play log files to restore to a specific point-in-time. Database restores, generally speaking, take two forms:

  • Recovery – Overwrites the original database or VM and is used for true data recovery scenarios.
  • Data Delivery – Maps servers to clones of databases or spins up VMs for non-recovery use cases (test-dev, etc.).

Restore becomes more complicated with a SQL failover cluster configuration. By default, the same disk signature will be reused which prevents you from restoring databases back to the original location. For this reason, it only works for limited disaster situations.

If a database clone is presented to a server and duplicates an existing disk signature, it cannot be connected to the server node because it creates a conflict.

To successfully map restored databases back to the original location while the original database is still running, you have to first map the restored databases to a different server (“proxy application server”), update the disk signature to avoid conflicts, and then map the newly signatured copy to the cluster.

This recovery method is effective but manually re-signaturing is complex, time-consuming and error-prone. This is not something you want during a recovery scenario when time to service is critical. It also impedes any efforts that involve data delivery. How can this be made easier?

ECX can orchestrate and automate this type of recovery efficiently and quickly, while allowing you to set up your recovery plan using a user-friendly web interface.

ECX automatically maps a clone to a proxy server. The proxy server updates the disk signature to avoid conflict.

ECX then maps the updated disk image to the SQL cluster, successfully completing the recovery process.

In the next section, you will see a step-by-step procedure to restore Microsoft SQL Server databases in a different location using ECX.

Configuring ECX

Open ECX from a supported web browser. From the navigation pane, click the Configure tab. Ensure you are seeing Sites & Providers and from the list under Provider Browser, right-click Application Server > Register.

In the Register Application Server dialog, click SQL and you will see a new dialog for Register Application Server (SQL). Fill in the required fields for each node of your Microsoft SQL Server cluster. You may select from either a physical server or a virtual machine hosted on VMware vCenter. Click OK to close the dialog.

Click the Jobs tab. By default, an Inventory Job starts as soon as you register the SQL Server. The Inventory Job will catalog the SQL cluster and identify the location(s) of the database files, log files, etc.

In the tree pane, go to All Jobs > Database > SQL, right-click it and you will see three options: Backup, Restore and Inventory. Click Backup and you will see the New DB/FS Backup Job dialog. Choose the template Standalone and Failover Cluster.

From the tree pane, expand Sites to see the database instances and databases that you want to protect with snapshots. You will also see a list of SLA policies in the right pane. These are created separately and applied to storage volumes, databases and/or VMs. Select the SLA policy to associate with the database(s), name this new backup job and click Create Job. This closes the dialog and starts the job immediately.

Go back to the Jobs tab, ensure SQL is selected and you will see a list of jobs for Microsoft SQL Server in the right pane. Click History in the bottom pane and click the latest job which usually is the one you created and started in the previous step. In the job window, you will see the ECX server orchestrating the backup job with your storage services (e.g. Pure Storage FlashArray) and Microsoft SQL Server databases. Ensure the backup job status changes into COMPLETED and close the window.

Now that your backup job has completed, you’ve got a SQL-aware snapshot you can use to restore from a data loss event, or you can use it to deliver copies for other uses. Let’s walk through the recovery process.

Go back to the Jobs tab, right-click SQL and click Restore. Choose the application Microsoft SQL Server (Standalone and Failover Cluster) and the template Instant Database Restore. In the New DB/FS Restore Job dialog, first ensure Source is selected in the left pane and from the Application Browse pane, select the sites, database instances and databases you want to restore.

Since ECX catalogs and tracks all copies, it knows where your copies are. Any available copies will be displayed, including remote site copies if you set up a replication policy. Click Copy from the left pane and select the copy location you wish to use.  By default, the latest snapshot will be used to restore your data, but click Choose a specific version and you can choose any snapshot that was created on a specific date and time.

Third, click Destination from the left pane, select a site and a database instance. As seen in the Select a Windows Server to resignature LUNs section, ECX allows you to restore your databases in the same Microsoft SQL Server node (as well as different nodes) through a proxy application server. Although you may rename your restored database to avoid conflict with the original, ECX will reuse the original name by default. Click Advanced Options for additional options such as overwriting existing databases and specifying a mount point. Click OK, the dialog closes and you will see the restore job in the Activity list in the bottom pane.

Once it has done, the status of the job changes into RESOURCE ACTIVE. Click this job and you will see the restored databases have mounted and their LUNs are mapped to the target destination Microsoft SQL Server nodes. Log into the target Microsoft SQL Server and you will see the restored database(s).


As you have seen, ECX allows you to protect your Microsoft SQL Server databases with ease while offering many practical options to meet your needs in various situations. For more information about how ECX can protect your business assets, refer to the product documentation or contact our specialists at Catalogic Software. And if you want to see some live demos of ECX on Pure Storage, you can find them here.

Read More
09/04/2019 0 Comments

Let us show you around

Data ProtectionData ManagementOpen VM BackupNetApp Ransomware ShieldNetApp File Catalog