Building a new Lustre Filesystem

Here are the quick and dirty steps to create a new Lustre filesystem for testing purposes. I use this at times to test out commands and test benchmarking tools, not to test performance, but to ensure they operate correctly.

This is a simple test environment on a single system with a single physical disk. Lustre is designed for scalability so these commands can be run on multiple machines and across many disks to ensure that a bottleneck does not occur in larger environments. The purposes of this is to generate a working Lustre filesystem for testing and sandbox work.

This set of directions assumes you have compiled and installed both the Lustre kernel and the Lustre userspace bits. Check my previous blog posting for how to complete those items if necessary. This also assumes that you have a spare physical disk that can be partitioned to create the various components of the filesystem. In the example case below I created the filesystem within a Xen virtual machine.

1) Create a script to partition the disk that will be used for testing (using /dev/xvdb for example purposes)
#!/bin/sh
sfdisk /dev/xvdb << EOF
,1ooo,L
,1000,L
,2000,L
,2000,L
EOF

2) Format the MGS Partition
- mkfs.lustre --mgs --reformat /dev/xvdb1
- mkdir -p /mnt/mgs
- mount -t lustre /dev/xvdb1 /mnt/mgs

3) Format the MDT Partition
- mkfs.lustre --mdt --reformat --mgsnid=127.0.0.1 --fsname=lusfs01 /dev/xvdb2
- mkdir -p /mnt/lusfs01/mdt
- mount -t lustre /dev/xvdb2 /mnt/lusfs01/mdt

4) Format the First OST Partition
- mkfs.lustre --ost --reformat --mgsnid=127.0.0.1 --fsname=lusfs01 /dev/xvdb3
- mkdir -p /mnt/lusfs01/ost00
- mount -t lustre /dev/xvdb3 /mnt/lusfs01/ost00

5) Format the Second OST Partition
- mkfs.lustre --ost --reformat --mgsnid=127.0.0.1 --fsname=lusfs01 /dev/xvdb4
- mkdir -p /mnt/lusfs01/ost01
- mount -t lustre /dev/xvdb4 /mnt/lusfs01/ost01

6) Mount the client view of the filesystem
- mkdir -p /mnt/lusfs01/client
- mount -t lustre 127.0.0.1@tcp0:/lusfs01 /mnt/lusfs01/client

At this point you should be able to do an ls, touch, rm or any other standard file manipulation command on files in /mnt/lusfs01/client.

Building Lustre 1.6.5.1 against the latest Redhat Kernel

I was at a customer site this week and had the need to build Lustre 1.6.5.1 against the latest kernel from Redhat, 2.6.18-92.1.13. Being this process has multiple steps, I thought I would document it so that others do not have to reinvent the wheel.

1) Prep a build environment
- cd ~
- yum install rpm-build redhat-rpm-config unifdef
- mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
- echo '%_topdir %(echo $HOME)/rpmbuild' > .rpmmacros
- rpm -i http://mirror.centos.org/centos/5/updates/SRPMS/kernel-2.6.18-92.1.13.el5.src.rpm
- cd ~/rpmbuild/SPECS
- rpmbuild -bp --target=`uname -m` kernel-2.6.spec 2> prep-err.log | tee prep-out.log

2) Download and install quilt (quilt is used for applying kernel patches from a series file)
- cd ~
- wget http://download.savannah.gnu.org/releases/quilt/quilt-0.47.tar.gz
- gunzip quilt-0.47.tar.gz
- tar -xvf quilt-0.47.tar
- cd quilt-0.47
- ./configure
- make
- make install

3) Prepare the Lustre source code
- Download from http://www.sun.com/software/products/lustre/get.jsp
- mv lustre-1.6.5.1.tar.gz /usr/src
- gunzip lustre-1.6.5.1.tar.gz
- tar -xvf lustre-1.6.5.1.tar

4) Apply the Lustre kernel-space patches to the kernel source tree
- cd /root/rpmbuild/BUILD/kernel-2.6.18/linux-2.6.18.x86_64/
- ln -s /usr/src/lustre-1.6.5.1/lustre/kernel_patches/series/2.6-rhel5.series series (there are several diffrent series files in the series dir, choose the one closest to your environment)
- ln -s /usr/src/lustre-1.6.5.1/lustre/kernel_patches/patches patches
- quilt push -av

5) Compile a new kernel from source
- make distclean
- make oldconfig dep bzImage modules
- cp /boot/config-`uname -r` .config
- make oldconfig || make menuconfig
- make include/asm
- make include/linux/version.h
- make SUBDIRS=scripts
- make rpm
- rpm -ivh ~/rpmbuild/RPMS/kernel-2.6.18prep-1.x86_64.rpm
- mkinitrd /boot/initrd-2.6.18-prep.img 2.6.18-prep
- Update /etc/grub.conf with new kernel boot information

6) Reboot system with new, patched kernel

7) Compile Lustre with the new kernel running
- cd /usr/src/lustre-1.6.5.1
- ./configure --with-linux=/root/rpmbuild/BUILD/kernel-2.6.18/linux-2.6.18.x86_64
- make rpms (Build RPMs will be in ~/rpmbuild/RPMS)

8) Install the appropriate RPMs for your environment

Getting Ahead in IT

I speak with folks in Information Technology (IT) regularly that tell me how hard it is to get ahead in IT and that there are too few opportunities for promotion in IT. I hear this from folks at all levels including architect, developer, administrator and team management. While getting ahead in IT can be difficult, it is not impossible. IT is often the last organization within a company that Human Resources (HR) considers when developing career paths, career training and mentoring plans. There are a variety of things that people in IT can do to ensure they get noticed by management and advance as a result.

Have a career path
First and foremost in IT it is important to have a career path, know what you want to get out of your career in the immediate future, in 5 years, and beyond. This will allow you to strategically pick projects that increase your ability to meet these goals. This plan will also allow you to speak with your manager, HR and other company leaders about training, mentoring, and other activities to increase your skill set.

A career path is not necessarily a goal to be promoted or to obtain a title that you would like. While it can be those things, it is more often targets for growth in technical capability, a goal to become management, or a goal to develop skills in one role that will enable you to deliver more efficiently in another role. Each of these goals requires a different focus on skills development, but all require the same open communication with your current management about your goals.

Focus on Business Needs
Second, being in IT does not preclude you from participating in the business. IT is an integral part of any business, and the IT folks that excel are the ones that understand how IT can help the business grow and be more efficient. Working to establish an understanding of the companies core principals will allow you to suggest improvements in IT, as well as establish a value within the company.

Very seldom do IT folks participate in business focused meetings. This is a shame. This is a very good opportunity for IT folks to not only learn about a companies' operations, but it is also a good chance to suggest new and better methods that IT can provide for the core business of a company. The IT staff that take the initiative and participate in the business discussions will be the ones most noticed as new opportunities within the company open up.

Bring Forward Ideas
Being noticed is important in all careers, especially in IT where management often forgets how critical IT is to the success of a company. The easiest way to be noticed is to speak up, if you have an idea for improvements, be it a new tool, an improved process or a new piece of hardware, suggest it to management. If your immediate manger does not see the value, mention it next time you are talking to other managers. Management at all companies appreciate individuals that are proactive enough to suggest ways for improvement before being asked.

Set Boundaries
At high stress times when projects are due and deadlines are tight it may not seem like it, but managers will notice and respect you for holding to your principals. Setting boundaries is important in all jobs, especially in IT where often long hours are the norm and expected. Work with your management to let them know about outside obligations, either organizations or family. This will ensure your management is aware of other things you are involved in. Most importantly hold to those boundaries. It is normal to have a long evening when a project is due, or the systems are down. The important thing is to stick to your boundaries and do not allow one late evening to morph into constantly working excessively long days.

Setting boundaries will also help keep stress within a reasonable level. If you can keep your stress lower, you will not only be a more efficient employee, but you will be able to more effectively assist others and complete your own projects.

Foster an environment of personal development
Often times folks say that a work environment is created by management, this is only partially true. A dedicated staff member can also create an environment where others share, feel comfortable asking questions and learn. By letting others know you are available for questions or conversation, you are letting others know that you are willing to help others develop their skills and experience. This attitude can be used to influence management at all levels to formalize personal development plans.

Staying positive is an important component of developing an environment people want to work in. No one wants to be known as the angry employee. It can be tough to keep your cool at all times, but being the level headed employee goes a long way to making yourself approachable and creating a comfortable working environment.

Be a Team Player
Everyone is told from childhood to be a team player, but what does that mean in todays business environment? Being a team player is a combination of sharing the work load, accepting projects that benefit the team as a whole and ensuring information is shared for the benefit of the team.

Most importantly, especially for folks that manage teams and projects, do not pass along a task because you do not want to do it. The quickest way to get noticed is step up and complete the work that needs to be done, but would otherwise fall by the way side because folks do not find it interesting. You certainly do not want to make a career out of working on uninteresting projects, but picking up one now and again will not be a career killer and will get you some recognition.

The most efficient way to show you are a team player is to not work in a vacuum. As you and your team work on projects, solicit the input from other teams. Use there input to carefully evaluate your teams' assumptions and project goals, and make adjustments as necessary. This shows that you value their input and experience, and will ultimately enable you to create a better product for the company.

Document, document, document. In todays world where people regularly move roles and companies, it is critical to ensure that all tasks, no matter how trivial, are documented so that others can complete them. If your company has a wiki, use it. If your company does not have a wiki, put one online for others to use. The fastest way to develop mindshare within a company is to be the person who has contributed the most to internal repositories. By ensuring your tasks are carefully documented, you are telling company management you are not trying to force them into keeping you, you are telling them you care about the companies long term success and letting them know you can be moved to other roles and new staff can take over your tasks.

Moving On
Ultimately, some combinations of employees and companies will not mesh well. In times like that it is appropriate to look at opportunities elsewhere. There are a lot of companies in IT today, and often times a different one will provide you the opportunities you are looking for. But make sure that any time you are looking to move companies you closely assess why your current company is not providing what you need to meet your goals, and work to find a place that will assist in meeting those goals.

When looking for a new role or starting a new role, remember that these things take time. It can often take months or years to feel at home at a new company and really feel like you are a highly contributing member of the team. When looking for a new role, discuss with your potential manager how long staff have been at the company, what types of development opportunities they offer and how the team dynamics work. This will ensure that any job change is meaningful and a path to newer and better things.


IT is not the dead end that it is often made out to be. It is also not a simple process to make yourself known in a large pool of people and advance in IT. To succeed in IT you must have a clear set of goals for your career, and use those to develop a clear list of activities to meet those goals. Moving up is not an immediate process, but by committing the time to development and communication, you can let management know you are willing and capable to take on new challenges and meet your career goals in the process.

Succeeding in todays services driven IT market

The Information Technology (IT) space is undergoing a dramatic shift. This shift from hardware and software based sales processes will have a dramatic impact for those executives that manage IT vendors, those who sell IT solutions and those who market them. As hardware prices have fallen and hardware has become more commodity in nature, companies are focusing less time on purchasing hardware, and more on ensuring that their business needs are being met by their IT systems. Companies are beginning to realize that IT can be an enabler to ensure that their primary business is run as efficiently as possible. This realization is opening new markets around services, primarily highly complex consulting and integration focused services.

Service Definitions
To enable organizations to successfully meet these new services driven customer needs, there is a separation occurring within a lot of services organizations. This separation is usually along operational lines to enable each type of services delivery organization to effectively delivery value to their customer, in a scalable manner. Organizations can be broken into four distinct services delivery teams, Professional Services, Consulting Services, Managed Services and Support Services.

There are a variety of interpretations today about what types of offerings are available from a services organization, and how they are branded publicly. When speaking about Professional Services, I envision an organization that is focused on product delivery and integration. Professional Services are often the organization a vendor uses for deploying their hardware and software in customer environments. Professional Services personnel are often experts in a companies portfolio, as well as the products integration with other offerings on the market.

Consulting services are a higher caliber of Professional Services in my experience. Consulting Services are offerings around custom integration or custom development, either system or software. Consulting Services tend to be more complex deals that run longer, and do not necessarily have a hardware or commercial software component as Professional Services would.

Managed services are often used as a way to ensure a vendor has a long term presence at a customer site. Managed Services offerings are often provided to the customer to manage the on-site hardware and software that a customer has purchased, but does not have the staff to operate day-to-day. Managed Services are often long term agreements for a company to ensure a customers' IT operations are stable and managed per industry best practices.

Finally, Support Services. Support Services are typically the contracts that are purchased with hardware and software to entitle the owner to a clear path for product assistance. This is most often phone support and access to patches for the product for bugs and security vulnerabilities.

Each of these four offerings is distinct, they each have their own lifecycle, associated costs and required skill levels for delivery. It is important to distinguish the various service offerings when developing sales strategies, as well as delivery methodologies. Each one is a different type of purchase for the customer, and has different implications to the cost/benefit trade off analysis work that customers do when purchasing services.


Hardware is cheap
One large influence on the drive towards services and specifically Consulting Services is the drive towards cheaper hardware. Hardware today is based on standards and commodity parts that enable a larger number of vendors to sell the same capabilities and components. Because of this commonality around features, customers look mostly to price when comparing two similar pieces of hardware. The companies that strive are the ones that realize hardware is only a platform for running a business; the real value to companies in todays fast paced market is putting highly capable solutions on this common hardware to enable a customer to be more successful.

Many companies today rely on a predictable, regular refresh cycle for all hardware. This enables companies to position themselves to deliver solutions around managing services on top of this regular refresh. This refresh requires companies to ensure that data and applications are implemented in a way that when the next hardware refresh comes, the data and applications can be easily migrated. Often times customers do not posses the necessary staff in house to implement these types of software provisioning, they will turn to consulting organizations to implement integrated solutions around these refresh cycles.


Fixing Business Problems
Todays customers are looking more and more to IT as a way to enable their core business in a more efficient way. Customers are using data warehouses as a way to process vast amounts of data to ensure the business is being managed correctly, they are using customer resource management systems to ensure customer requests are handled efficiently and correctly the first time, and they are using mobile devices to connect remote workers to the office and get them information as soon as possible.

Often times companies do not have the necessary staff in house to both implement and manage todays complex solutions. Customers often must balance having too many versus too few staff, and often will lean towards fewer staff, and contracting the complex implementations and projects that require more time then staff immediately have.

Customers today will also look to outside services for guidance in business inefficiencies. Often customers see a value to outside input when reviewing legacy processes, this outside input can ensure that new processes are developed with an understanding of currently available technology and tools that can assist with driving productivity. These business efficiencies have a multitude of possible solutions including business intelligence tools with an associated data warehouse, a formal enterprise architecture program, automated provisioning of new services or automated software development assistance.

Business Intelligence
Todays businesses rely on information for making decisions, as well as reviewing previous decisions. in a systematic way. This information must be organized and have associated tools for reporting. Often times companies will look to outside firms to assist them with managing their sales information, forecasting, process assessment, manufacturing data and purchasing. Todays business intelligence solutions rely on expertise in these areas in data management, data mining, data cleansing and ultimately reporting using accurate and proven methods.

Enterprise Architecture
Enterprise Architecture is growing in popularity as companies look to formalize how business processes and company visions become IT systems. Todays TOGAF and the Zachman Frameworks are used by countless organizations to formally document the architecture that IT will follow for implementing systems, tools, software and support services. Few companies today have the expertise in house to develop a formal Enterprise Architecture program, and because of that will look to outside companies that have expert level knowledge and experience with the above frameworks.

Automated Provisioning
Speed is an important factor when doing business today. Companies that can rapidly adapt to change are more successful in meeting customer demands and needs. A companies IT systems are a critical component to all adjustments to market environments. By being able to more rapidly provision new services, or capacity for existing services, companies can ensure they are ready for this change. Automated provisioning ensures that minimal staff intervention is needed when bringing online new services; this both lowers the time to market as well as the costs associated with bringing new capabilities and capacity to market. Companies will often look for experienced outside assistance when developing automated provisioning systems, this outside experience can ensure that new services are brought online both efficiently and correctly.

Software Development
Software development can be a complicated orchestra including requirements gathering, architecture, development, internal testing, and finally customer testing. Companies will often look for external assistance with developing unit tests and automated regression testing environments. Outside resources can provide a unique perspective on the development and testing process because of there disconnect from the entire development process, they are able to focus all efforts on testing for defects and usability.


Selling to the decision makers
This shift in IT purchases from hardware to services has a dramatic impact on the sales process for vendors, particularly those that sell both hardware and services. As more and more IT solutions are purchased that are directly tied to company objectives, fewer purchases will be made by the managers and staff implementing the solutions. More and more large IT purchases are being made by a companies executives including the CIO, CFO, CTO and COO. These individuals are no longer focusing on the technology behind the products, they are looking to vendor solutions and offerings as a way to increase productivity, increase output, and to better understand and manage their business.

When selling services and solutions today, sales teams must articulate to potential customers the immediate and long term costs of solutions, and how those costs will directly affect the bottom line of the business. The cost of individual servers, licenses and data centers are no longer looked at with the level of scrutiny they once were. Today purchasers are looking to the total cost of a solution in implementation costs, reoccurring costs, and upgrade costs; then comparing those costs to the measurable benefits once the solution is in place.


Considerations for solution based purchases
Now that we know who is making the decisions around big IT purchases and that hardware is a small piece of the equation, we need to understand what items influence these decisions. These will enable solution developers and sales teams to properly positions services within a customer organization to enable effective business solutions.

Life-cycle Support
Customers look at vendors to provide them solutions that will meet the immediate needs as well as provide paths as the company grows aver time. The paths can be upgrades, accommodation of new legal requirements, growth and manageability. As part of the sales process it is critical to communicate to the customer a vendors capabilities around not only solution implementation, but also upgrades, changes and solution support.

Solution Ownership
Todays companies look to do business with vendors that will own solutions from end to end. This ownership requires the vendors to have solid methodologies around product development, delivery, support and upgrades. This does not mean a vendor needs to develop all products in house, or have a software package for every customer, but vendors should be able to provide their customers a single point of contact for all phases of complex projects.

Solution Flexibility
Companies today want to ensure they are not locked in to any specific solution, be it hardware, software or a specific consultant. For a consultant to put in the extra effort to ensure a solution is properly documented and communicated to the customer, shows the customer the dedication to their business and helping them succeed.

Cost Management
When purchasing services related IT solutions, companies today are looking at more then the initial cost of the contract, or the options. They are looking at the benefits the company will see because of the solutions, these benefits can be more efficient operations, more customers, or simplified growth paths. It is critical that as part of the sales process the costs and benefits are both understood and communicated to the customer.


To be successful in todays IT market, vendors must focus on correcting customers business problems, and work to become a trusted adviser in their business operations. Customers today are looking for long term solutions to their IT needs that will ensure they are competitive and able to not only grow, but change as the market demands. This has caused a dramatic shift away from purchases focused strictly on servers and storage, but to purchases of solutions. These solutions must have definitive cost returns over time that allow management to clearly understand how their business will be positively impacted.