Presenting ECX 2.5, Our Biggest Release Yet!

By leveraging native copy services provided by storage, they validated our vision and value proposition even more. Within a few days of that announcement, we saw heavy interest from various storage vendors to add support for their storage arrays within our platform to provide “in-place” copy data management using storage snapshots, replication and zero-footprint clones.

The leading contenders were the Pure Storage FlashArray family and IBM Spectrum Accelerate family of Storage (a.k.a. XIV). Both platforms come with industry leading data reduction, deployment simplicity, and deliver superior performance with their flash offerings. Our copy data management features deployed on top of these platforms allow customers to squeeze every penny of value out of their investments into these great products. Pure Storage preferred to announce the partnership at VMworld and IBM preferred at IBM Edge, both only a couple of weeks apart. We’ve historically only been able to add support for one storage platform every major release, every 4 months. This time, with two willing and supportive partners, our adventurous engineering team took on the challenge to deliver support for both in the same release cycle. And we did it!

Releasing support for Oracle in ECX 2.4 showed us that the value of copy data management is exponentially higher for applications. A leading bank told us that for every 20 TB of Oracle data that they have in production, they provision over 300 TB (!!) of storage to accommodate all the copies they need for performance testing, regression testing, analytics, staging, backups, etc.

In ECX 2.4, we ingeniously came up with an agentless method (using runtime code injection) to provide storage snapshot support for Oracle. The method had near-zero impact on database servers enabling creation of much more frequent copies. Use of zero-footprint storage clones ensured one can provision many copies of these databases with no additional storage utilization. And we found a way to record our copies within RMAN and weave together data masking workflows. However, Oracle was always a proof of concept, the first of many applications we intended to support. Next on the list for ECX 2.5 was expanding the same support for Microsoft SQL Server, and we did it!

All data management solutions are traditionally handled through an operations team. We recognized a shift in more organizations moving to a self-service model – letting business units take care of their own needs while still retaining central control within IT. In order to aid this, we wanted to expand our RBAC capabilities to include fine grained control over not just policies, workflows and reports, but also individual VMs and databases. And we did it!

And there is an even bigger transition of organizations moving away from a model of people running systems to a model of systems running systems. This is an essential part of continuous integration and continuous development (CICD), a DevOps model. There are popular tools such as Jenkins and Puppet that specialize in this world of spinning up test servers with static images, but struggle to mirror and keep up with production datasets. An even bigger struggle involves transitioning of these test setups into production, or dealing with data clean up. This is our bread and butter! We saw a mutual fit where we could build plug-ins for DevOps tools like Puppet and Jenkins to leverage our workflows, and we did it!

If that wasn’t enough work put into ECX 2.5, late in the release cycle a prestigious storage vendor (still can’t name names) showed interest in OEM’ing our solution. In record time, we refactored our framework to allow for white-labeling the solution. While we did deliver above and beyond all our targets in ECX 2.5, perhaps the best compliments came from a major player’s decision to put their name on our product.

It’s really quite a release!

Read More
10/20/2016 0 Comments

Pure Storage Snapshot Management Solution Now Released!

We’re happy to report that we’ve released ECX v 2.5 which contains support for the Pure Storage FlashArray family, both the //m series and the previous FA-400 generation. The reception of our Pure Storage support has been nothing short of amazing. Let me explain.

Since we first announced our plans to support Pure Storage back in August, Catalogic attended three major trade shows in the span of five weeks (that was a lot of time in Las Vegas!)  We had hundreds of discussions with Pure Storage customers and prospects, and the excitement level was palpable. The conversations revolved primarily around two main themes.

The first is that ECX provides an easy-to-use, workflow-based policy engine for Pure Storage FlashRecover snapshot and replication processes. Users were eager for a tool that could save them a lot of day-to-day maintenance (no more scripts!) and give them easy set-and-forget snap and replication management, along with push-button restores and comprehensive reporting and visibility. Our recent additions of agentless application support for Oracle and SQL Server really rounded out the solution.

But an area of even more excitement was what we can do for dev-test environments. In one conversation I had with a storage infrastructure manager, I brought up the subject of dev-test. His immediate reaction was, “Oh, those guys are killing me!” Just so! And that’s where ECX can help.

With ECX you can deploy copies – copies of data, or entire virtual environments – at the push of a button, or automatically on a scheduled basis. For example, if a dev-test team needs fresh copies of Oracle data, with sensitive data fields masked for security and compliance, ECX can have those delivered in place in minutes. How much time does this save? Well it varies across IT departments, but we’ve been hearing over and over that setting up dev-test takes several weeks at best. How about doing it in five minutes?

It’s these kinds of dramatic time savings that really make ECX a must-have tool for all-flash storage environments. Another person I spoke to had made the decision to move to Pure Storage from a competitive all-flash system that had proven problematic and disappointing. He was very happy with his storage choice, but he dreaded the weeks and weeks of internal software development effort it was going to take to automate his dev-test environment, to say nothing of maintaining that setup of countless scripts and multiple different tools across storage, compute, networking and application layers. When he saw that Catalogic could do everything he wanted with a simple GUI interface and no scripts, he was completely delighted. I’ve been in the software game for a long time, and I’ve rarely seen people so enthused for a solution as they are for this one.

Want a quick overview of what Pure Storage and Catalogic can do together? Take a look at our Copy Data Management for Pure Storage solution sheet and for more resources, visit our Pure Storage solution page, which is also where you’ll find new content as it arrives.

Speaking of new content, we’ve got a great new technical white paper about using Catalogic and Pure Storage in an Oracle database environment. It gives you all the step-by-step instructions you need to create application-consistent snapshots, and what’s more, to make easy use of them, whether for data recovery or for value-added use cases like dev-test and reporting. Download it now!

Read More
10/19/2016 0 Comments

Does Anyone Want to Run Their VM on My Enterprise? Dealing with VM Sprawl.

Catalogic ECX for cataloging the enterprise (eating our own dog food!).

In addition to in-place copy data management or data protection, ECX also provides search and analytics using a rich meta-data catalog. One of the cool reports in the mix is the VMware VM Sprawl report. I decided to give a try. Hooray!  The right prescription for the sickness – an interactive sprawl report with the right set of filtering capabilities. The report showed that there were several VMs up and running for more than a year and nobody was using them! Several VMs were in a powered-off state and many of them were consuming a lot of storage space. The report was clearly exposing a sprawl problem. We were complaining about an underperforming environment and actually it was underutilized! You can see the kind of data the report returned in the screen shot below (we’ve blocked out the machine names for security).


If that’s a little difficult to read, here’s a close up of the relevant age data:


If you ever moved homes, you know how fun (;-)) it can be, especially if you were not doing a good job of throwing away stuff or donating it. When you start packing, you may see several boxes untouched in the closet for years. This is the right moment to ask yourself: do I need to carry these again to another place? Those are not easy decisions, especially if there is any sentimental value attached to it. If you are not sure, then you may send them to a storage place as insurance. If you read up to this point, I can bet that you had this experience at least once. Did you make the right decisions?

Cleaning up the vSphere environment is no different. If you need to buy several tools to do the job, you may be out of budget soon. But ECX is your buddy. You do not have to look anywhere else. One tool does it all, and it will help you to reduce your storage costs with its in-place copy data management capabilities and with its data protection capabilities. Isn’t that great? What else do you need?

Okay, I got my prescription and now I need medicine. Why not turn to ECX again?  First thing I noticed in the report was that there were some clear patterns in the VM names. So I decided to do a google-like search in ECX using the Search window. Wow! ECX returned all the VMs I was interested in, so know I knew how many there were and where they were located.

Another cool feature is that I can find historical data in few clicks.  Some of the VMs were created by some of my previous colleagues in the company. So that was an easy call for cleanup. The next set of VM’s, I wasn’t sure. So I sent a nice note to my colleagues and they helped me to identify systems that could be removed. Again another good chunk of VMs left and not sure whether we need it back. This was exactly the same moving experience I was talking earlier – we tend to keep things around even it was left in the closet untouched for years! I decided to move them and reclaim those resources from the expensive storage. Initially, I considered using in-place copy of these VMs to a low-end storage tier and then deleting them from the production LUNs.  On second thought, I might be pushing the problem downstream. Also I do not want to use tapes (sorry, no offense). What else might be a better choice? Cloud!

ECX has a native integration with AWS cloud gateway. Another cool feature from the 2.4 release! I simply created a copy policy with a few clicks and those virtual machines landed in the cloud in few hours! Now I have additional 40TB of storage space and all the running virtual machines have more juice. Now we have an underutilized vCenter. So does anyone want to run their VM on my enterprise?

Ranjith Govindan is a Catalogic Senior Manager, Research and Development

Want to have ECX report on your VMware infrastructure? You can download a copy and try it for yourself for 30 days.

Read More
10/10/2016 0 Comments

Copy Data Management and Jenkins Continuous Integration

Jenkins is a continuous integration solution used by many development organizations to automate software building, testing, and deployment. Copy Data Management (CDM) is a term used to describe software solutions that manage the proliferation of storage, virtualization and application snapshots within an operations context. Solutions in this category make it easy to re-use production copies of systems-of-record for development and testing purposes.

DevOps fosters a culture that bridges operations and development disciplines toward streamlining value creation for the business and end user.  This post will describe how Catalogic’s ECX CDM solution integrates with Jenkins, providing a powerful solution that produces copies of production environments updated with development’s latest code modifications within minutes.  

DevOps Use Case

As a QA member, I would like to have a copy of a live production system with the newest set of components from development to perform testing. Ideally, the creation of this updated production copy should be initiated by a QA or development member in a self-service/on-demand manner.  The entire process should take no more than a few minutes. Here is the activity that is expected to be carried out:

A self-service portal will facilitate the selection of a specific production copy version that will be leveraged for testing the latest changes from development. The latest changes from development may be organized as labeled source control repository branches. The developer or QA engineer will select a particular source code version from a repository along with a version of a production environment and then perform an operation to spin up the copy.  A Jenkins project will  be executed that will perform the following activity:

  • Pull the specific source code from the source code repository (git or SVN)
  • Transfers the source to a build machine, which runs a Jenkins slave agent
  • Run the build to produce the binaries and executables
  • Deploy the copy of a production environment
  • Update the production environment with the binaries and executables
  • Report back to the developer or QA engineer with job status

Jenkins-CDM Plugin

Integration between ECX-CDM and Jenkins makes the above use case possible. This integration takes the form of a Jenkins plugin that participates in the life cycle of a Jenkins Continuous Integration job which calls into ECX-CDM via its REST APIs.  The following is a high-level illustration of the plugin and integration:



The diagram depicts the Jenkins workflow leveraging the plugin to spin up a copy of a production environment so that it can push updates to it. The plugin’s GUI facilitates the following:


  • Allows one to define, through the Jenkins user interface, login credentials to ECX
  • Select a particular ECX policy to run that will produce a copy of a production environment

In Conclusion

The following diagram depicts the workflows managed by ECX-CDM and Jenkins leveraging the Jenkins ECX-CDM plugin.


For more details on how Catalogic can help you with DevOps, visit our DevOps solution page. Catalogic can also help if you are using more traditional Dev/Test methods

Jay Hamilton is Senior Manager of Software Research and Development at Catalogic. He has blogged previously on “Delivering Continuous Value to Customers via Agile Software Development and The Marketplace.

Read More
10/04/2016 0 Comments

Let us show you around

Data ProtectionData ManagementOpen VM BackupNetApp Ransomware ShieldNetApp File Catalog