Making Agile BI Work
In our recent blog entry we described Agile BI and why it is very important for organizations to apply it. In this entry we focus on what makes a successful Agile BI Project.
Proper expectation management
De-mystifying Agile Bi means slavishly practicing expectation management. Expectation management establishes the foundation of success for an Agile BI project. Remember Agile BI is not the silver bullet; just an effective methodology of work. From a technical perspective don’t try to “boil the ocean” and join billions of records with billions more records and bring the entire result set back in sub-second time. To manage your business users’ expectations, closely work with them to identify the appropriate use cases is important. Succinctly and clearly define what they can expect and the explicit process to be followed during the Agile BI project.
Appropriate use case
Identification of the appropriate use case or cases should focus on finding the “low hanging fruit.” What I mean by that is find a problem that can be solved with Agile BI and still deliver a LOT of value to the business. Some IT-centric projects may be “cool” but if there’s not real business benefit to your business users then it’s not going to catch their eye. Understand and analyze their issue, try to figure out how much it’s costing the business because of this challenging pain, then prioritize and deliver. Ensure that you know what success is and establish goals for Agile BI.
As we all know the success of BI projects hinges on the availability of needed and clean data. Profile early and often! If I had a dollar for every time one of my customers or prospects told me “I know my own data and we don’t have any data quality issues,” I’d be rich. Don’t fall into that trap. Don’t be THAT guy! Profile early, often, don’t make assumptions about your data and let the data speak for itself! The last thing you want to do is go through an exercise for an Agile BI project, spending a lot of time working on build the perfect reports and dashboards, understanding what the data is and how it helps solve a critical set of problems for the business…only to deliver reports with duplicate data, multiple formatted data, or worse – NULL data values! Treat it like a true data warehouse BI initiative and profile to understand what data quality issues are really lurking at the data sources. Also, be prepared to have to embed data quality rules in your data integration layer as well as in your virtualization layer.
Preparation of environment
This is my personal pet peeve: environments that are not prepared for software to be installed and configured. You cannot be agile if you have to wait for your systems, your database administrators, your systems administrators, or your business systems analysts. Work with your vendors as well as your systems integrators to help ensure that you have not just the right minimum requirements, but the best requirements for any tests that you may want to try to run if you are evaluating Agile BI software or doing a pilot. I can’t recall how many times I had customers or prospects “assure me” that their environment was ready when they were behind in their project. It was typically those folks who ran into issues with their environments when we did arrive onsite. Here are some issues to focus on and resolve to prepare for your agile BI project:
- Server – Size it appropriately based on vendor and/or SI feedback
- Server IDs and passwords – Ensure that you have all the necessary ID’s and credentials with the proper permission and privileges
- Databases – Ensure that you have all necessary database schemas set up and ready to go to ensure a smooth installation
- Database ID’s and passwords – Ensure that you have all the necessary database ID’s and proper credentials so that the software fails. Many times a software package has a robust metadata repository. During installation the database ID will try to create tables, views and insert metadata.
- Access to critical resources – Ensure that you have access to database administrators and systems administrators that may be needed during a critical installation phase. Ensure that business systems analysts are available and working as part of the Agile BI team to help give critical feedback.
Find the right balance
The essence of Agile BI is making smart trade-offs seeking the balance of success. Here are some tips:
- Sprint Activities vs. Finish Line
You typically will have two main types of activities during your Agile BI project: sprint activities and end goal or finish line activities. Sprint activities are typically viewed as somewhat of “throw away” activities – profile to take a peek at the data as soon as possible, build some quick views or virtual table to expose the data to the business so they can see what it’s really like and what is valuable. Finish line or end goal activities could be a finished data model for an Operational Data Store or Data Warehouse, a final dash board or set of reports. These reports can be based on a virtual layer so that they may be the end goal but they could be accessing some intertwined sprint activities. Sprint deliverables need to take into consideration the end state in order to minimize “throw-away.” Not all aspects of BI projects lend themselves to Agile methods. For example, data modeling needs to have a view of the end in mind so each sprint is an exercise in refinement since it is more feasible to modify ETL and reports.
- Functional Team Members (business, QA, developer, etc.) all are Critical
One of the main traits of an Agile BI project is that it brings together resources from multiple departments: business systems analysts, QA analysts, data integration developers, and BI developers. You cannot build reports, integrations, or virtual tables in a vacuum without the continuous presence of the business. They know the valid values of the data, what the data means and is supposed to mean, and they ultimately can tell IT where to help them find the nuggets in all of their data assets.
- Scope and Time – What can be Delivered in a Tight Cycle
Adjust the scope to fit the time box but be certain the integrity of the use is maintained. Just because you may start to embark on an Agile BI project and it may have a virtual layer, don’t try to create every single slice of business in your virtual data model. Don’t boil the ocean… Prioritize! Find the highest value, learn the software and new concepts for your initial Agile BI project, build project momentum, then iterate through again and find newer pieces of business that you could expand upon your now successful virtual data model. Take a deliberate phased approach – start with your current high value business slices – but also start to plan out the next phases.